Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On Feb. 27, 2023, 6:21 a.m., Madhan Neethiraj wrote: > > Andrew - the changes look good. As discussed earlier in this review, DB > > changes to remove unused columns should be done only in next major release > > - 3.0 i.e., in master branch. I suggest to split this patch into two parts: > > 1) Java changes to remove references to unused fields. This patch should > > be committed in both ranger-2.4 and master branches > > 2) DB changes to remove unused columns. This patch should be committed > > only in master branch > > Andrew Luo wrote: > Thanks - (1) is in this patch, and (2) is in thie patch but I've modified > db_setup.py to skip the patch (by skipping patches with SKIP at the end of > the ID) and also commenting out the contents of the patch. I wanted to do > this to ensure the patch is there for reference (the only difference is that > on master we will rename 061SKIP-... to 061-... and uncomment the contents of > the patch). If you prefer to remove it entirely - that's fine also, but we > will have to make sure nobody uses the 061 patch number on the 2.4 branch. > > Madhan Neethiraj wrote: > Thanks for the details of db_setup.py skipping DB updates in 2.4 branch; > I missed this. > > Considering 2.4 will only have updates in Java classes, is there a need > reserve a patch number (like 061)? DB patch can be merged in the master > branch with the most recent patch number. At the time of release from master > branch, the patch number might need to be revisited if any maintanence > release (2.5/..) ended up using that patch number. > > I suggest splitting this patch into 2 parts: > - Java class changes: to be merged in master and ranger-2.4 branches > - DB schema changes: to be merged in master branch only Okay - the original review you suggested to keep it in the "optimized" scripts (for new installations) while not having the patch on upgrade to preserve HA deployments. I think in this case it's ideal to reserve a patch number for this since if we don't reserve a patch number, there won't be any indication in the x_db_version_h that these columns will be removed, and then when upgrading to 3.x it will re-run this patch. It should still work fine but it's less efficient/ideal. Let me know what you prefer (or if you think we should remove the DB changes entirely from 2.4, including the "optimized" scripts) - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/#review225220 --- On Feb. 24, 2023, 1:23 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74301/ > --- > > (Updated Feb. 24, 2023, 1:23 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > We previously tried this commit and it caused a few issues (some typos on SQL > Server and SQL Anywhere, as well as upgrade scenarios - however this has been > fixed). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 9a79fe8ad > > security-admin/db/mysql/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > fd6cec9a7 > > security-admin/db/oracle/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 4d5a8cedf > > security-admin/db/postgres/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 3ed2a5b9c > > security-admin/db/sqlanywhere/patches/061SKIP-drop-audit-columns-from-policy
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On Feb. 27, 2023, 6:21 a.m., Madhan Neethiraj wrote: > > Andrew - the changes look good. As discussed earlier in this review, DB > > changes to remove unused columns should be done only in next major release > > - 3.0 i.e., in master branch. I suggest to split this patch into two parts: > > 1) Java changes to remove references to unused fields. This patch should > > be committed in both ranger-2.4 and master branches > > 2) DB changes to remove unused columns. This patch should be committed > > only in master branch Thanks - (1) is in this patch, and (2) is in thie patch but I've modified db_setup.py to skip the patch (by skipping patches with SKIP at the end of the ID) and also commenting out the contents of the patch. I wanted to do this to ensure the patch is there for reference (the only difference is that on master we will rename 061SKIP-... to 061-... and uncomment the contents of the patch). If you prefer to remove it entirely - that's fine also, but we will have to make sure nobody uses the 061 patch number on the 2.4 branch. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/#review225220 ----------- On Feb. 24, 2023, 1:23 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74301/ > --- > > (Updated Feb. 24, 2023, 1:23 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > We previously tried this commit and it caused a few issues (some typos on SQL > Server and SQL Anywhere, as well as upgrade scenarios - however this has been > fixed). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 9a79fe8ad > > security-admin/db/mysql/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > fd6cec9a7 > > security-admin/db/oracle/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 4d5a8cedf > > security-admin/db/postgres/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 3ed2a5b9c > > security-admin/db/sqlanywhere/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > ca8f7da1f > > security-admin/db/sqlserver/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/scripts/db_setup.py 24502f4fb > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 6cc3509d8 > security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java > 421b2312d > > security-admin/src/main/java/org/apache/ranger/biz/SecurityZoneRefUpdater.java > 4cfe62701 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8dfb92833 > security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefGroup.java >
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On Feb. 9, 2023, 12:16 a.m., Madhan Neethiraj wrote: > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > > Line 1433 (original), 1397 (patched) > > <https://reviews.apache.org/r/74301/diff/3/?file=2274595#file2274595line1433> > > > > Columns create_time/update_time/added_by_id/upd_by_id can be removed > > from following tables as well: > > - x_security_zone_ref_resource > > - x_security_zone_ref_service > > - x_security_zone_ref_tag_srvc > > - x_security_zone_ref_user > > - x_security_zone_ref_group > > - x_role_ref_user > > - x_role_ref_group > > - x_role_ref_role > > > > Also, I suggest to remove column guid from all _ref tables as they are > > not referenced from Java entity classes. Fixed and updated - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/#review225171 --- On Feb. 24, 2023, 1:23 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74301/ > --- > > (Updated Feb. 24, 2023, 1:23 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > We previously tried this commit and it caused a few issues (some typos on SQL > Server and SQL Anywhere, as well as upgrade scenarios - however this has been > fixed). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 9a79fe8ad > > security-admin/db/mysql/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > fd6cec9a7 > > security-admin/db/oracle/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 4d5a8cedf > > security-admin/db/postgres/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 3ed2a5b9c > > security-admin/db/sqlanywhere/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > ca8f7da1f > > security-admin/db/sqlserver/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/scripts/db_setup.py 24502f4fb > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 6cc3509d8 > security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java > 421b2312d > > security-admin/src/main/java/org/apache/ranger/biz/SecurityZoneRefUpdater.java > 4cfe62701 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8dfb92833 > security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefGroup.java > 22b944791 > security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefRole.java > 30867e28d > security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefUser.java > a5b17f716 >
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/ --- (Updated Feb. 24, 2023, 1:23 p.m.) Review request for ranger and Pradeep Agrawal. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). We previously tried this commit and it caused a few issues (some typos on SQL Server and SQL Anywhere, as well as upgrade scenarios - however this has been fixed). Diffs (updated) - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 9a79fe8ad security-admin/db/mysql/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql fd6cec9a7 security-admin/db/oracle/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 4d5a8cedf security-admin/db/postgres/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 3ed2a5b9c security-admin/db/sqlanywhere/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql ca8f7da1f security-admin/db/sqlserver/patches/061SKIP-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/scripts/db_setup.py 24502f4fb security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 6cc3509d8 security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 421b2312d security-admin/src/main/java/org/apache/ranger/biz/SecurityZoneRefUpdater.java 4cfe62701 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefGroup.java 22b944791 security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefRole.java 30867e28d security-admin/src/main/java/org/apache/ranger/entity/XXRoleRefUser.java a5b17f716 security-admin/src/main/java/org/apache/ranger/entity/XXSecurityZoneRefGroup.java 0ae6b2ffc security-admin/src/main/java/org/apache/ranger/entity/XXSecurityZoneRefResource.java 3d7197a16 security-admin/src/main/java/org/apache/ranger/entity/XXSecurityZoneRefService.java a2cacc674 security-admin/src/main/java/org/apache/ranger/entity/XXSecurityZoneRefTagService.java c67a4648d security-admin/src/main/java/org/apache/ranger/entity/XXSecurityZoneRefUser.java 4af242fd3 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 691ab52b3 Diff: https://reviews.apache.org/r/74301/diff/4/ Changes: https://reviews.apache.org/r/74301/diff/3-4/ Testing --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Also searched the source code and JPA queries xml for other references to these fields Thanks, Andrew Luo
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: (was: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch) > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-and-guid-fields-from-XXPoli.patch > > Time Spent: 10m > Remaining Estimate: 0h > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-and-guid-fields-from-XXPoli.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-and-guid-fields-from-XXPoli.patch > > Time Spent: 10m > Remaining Estimate: 0h > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4107) Upgrade EclipseLink
[
https://issues.apache.org/jira/browse/RANGER-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692787#comment-17692787
]
Andrew Luo commented on RANGER-4107:
[https://reviews.apache.org/r/74318/]
[https://github.com/apache/ranger/pull/221]
> Upgrade EclipseLink
> ---
>
> Key: RANGER-4107
> URL: https://issues.apache.org/jira/browse/RANGER-4107
> Project: Ranger
> Issue Type: Improvement
> Components: admin
> Reporter: Andrew Luo
>Priority: Minor
> Attachments: 0001-RANGER-4107-Upgrade-EclipseLink-to-2.7.12.patch
>
>
> See this comment from [~madhan] on RANGER-4104:
> I was able to reproduce failures in execution of named query
> {{XXAuthSession.getRecentAuthFailureCountByLoginId}} using a script that
> calls Ranger REST APIs from multiple threads simultaneously with invalid
> credentials. I notice at least 3 different types of failures:
> 1. ArrayOutOfBoundsException
> {noformat}
> [EL Warning]: 2023-02-22 20:36:40.8--UnitOfWork(729158064)--Exception
> [EclipseLink-6168] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.QueryException
> Exception Description: Query failed to prepare, unexpected error occurred:
> [java.lang.ArrayIndexOutOfBoundsException: 2].
> Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
> Caused by: Exception [EclipseLink-6168] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.QueryException
> Exception Description: Query failed to prepare, unexpected error occurred:
> [java.lang.ArrayIndexOutOfBoundsException: 2].
> Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
> Query: ReportQuery(name="XXAuthSession.getRecentAuthFailureCountByLoginId"
> referenceClass=XXAuthSession sql="SELECT COUNT(?) FROM x_auth_sess t0 WHERE
> (((t0.LOGIN_ID = ?) AND (t0.AUTH_STATUS <> ?)) AND (t0.CREATE_TIME >
> COALESCE((SELECT MAX(t1.CREATE_TIME) FROM x_auth_sess t1 WHERE (((t1.LOGIN_ID
> = ?) AND (t1.AUTH_STATUS = ?)) AND (t1.CREATE_TIME > ?))),? )))")
> at
> org.eclipse.persistence.exceptions.QueryException.prepareFailed(QueryException.java:1590)
> at
> org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:680)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:901)
> at
> org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:613)
> at
> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:867)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
> at
> org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
> ... 66 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.eclipse.persistence.expressions.ListExpressionOperator.getDatabaseStrings(ListExpressionOperator.java:75)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2196)
> at
> org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:99)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
> at
> org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:286)
> at
> org.eclipse.persistence.internal.expressions.RelationExpression.printSQL(RelationExpression.java:899)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
> at
> org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(Compoun
Review Request 74318: RANGER-4107: Upgrade EclipseLink to 2.7.12
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74318/ --- Review request for ranger, Kirby Zhou and madhan. Bugs: RANGER-4107 https://issues.apache.org/jira/browse/RANGER-4107 Repository: ranger Description --- RANGER-4107: Upgrade EclipseLink to 2.7.12 This fixes a number of issues observed with certain queries Diffs - pom.xml e402bcc5d Diff: https://reviews.apache.org/r/74318/diff/1/ Testing --- Tested with basic service create, policy create, etc. Should be a low risk upgrade Thanks, Andrew Luo
[jira] [Updated] (RANGER-4107) Upgrade EclipseLink
[
https://issues.apache.org/jira/browse/RANGER-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-4107:
---
Attachment: 0001-RANGER-4107-Upgrade-EclipseLink-to-2.7.12.patch
> Upgrade EclipseLink
> ---
>
> Key: RANGER-4107
> URL: https://issues.apache.org/jira/browse/RANGER-4107
> Project: Ranger
> Issue Type: Improvement
> Components: admin
> Reporter: Andrew Luo
>Priority: Minor
> Attachments: 0001-RANGER-4107-Upgrade-EclipseLink-to-2.7.12.patch
>
>
> See this comment from [~madhan] on RANGER-4104:
> I was able to reproduce failures in execution of named query
> {{XXAuthSession.getRecentAuthFailureCountByLoginId}} using a script that
> calls Ranger REST APIs from multiple threads simultaneously with invalid
> credentials. I notice at least 3 different types of failures:
> 1. ArrayOutOfBoundsException
> {noformat}
> [EL Warning]: 2023-02-22 20:36:40.8--UnitOfWork(729158064)--Exception
> [EclipseLink-6168] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.QueryException
> Exception Description: Query failed to prepare, unexpected error occurred:
> [java.lang.ArrayIndexOutOfBoundsException: 2].
> Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
> Caused by: Exception [EclipseLink-6168] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.QueryException
> Exception Description: Query failed to prepare, unexpected error occurred:
> [java.lang.ArrayIndexOutOfBoundsException: 2].
> Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
> Query: ReportQuery(name="XXAuthSession.getRecentAuthFailureCountByLoginId"
> referenceClass=XXAuthSession sql="SELECT COUNT(?) FROM x_auth_sess t0 WHERE
> (((t0.LOGIN_ID = ?) AND (t0.AUTH_STATUS <> ?)) AND (t0.CREATE_TIME >
> COALESCE((SELECT MAX(t1.CREATE_TIME) FROM x_auth_sess t1 WHERE (((t1.LOGIN_ID
> = ?) AND (t1.AUTH_STATUS = ?)) AND (t1.CREATE_TIME > ?))),? )))")
> at
> org.eclipse.persistence.exceptions.QueryException.prepareFailed(QueryException.java:1590)
> at
> org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:680)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:901)
> at
> org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:613)
> at
> org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:867)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
> at
> org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
> at
> org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
> at
> org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
> ... 66 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
> at
> org.eclipse.persistence.expressions.ListExpressionOperator.getDatabaseStrings(ListExpressionOperator.java:75)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2196)
> at
> org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:99)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
> at
> org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:286)
> at
> org.eclipse.persistence.internal.expressions.RelationExpression.printSQL(RelationExpression.java:899)
> at
> org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
> at
> org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExp
[jira] [Updated] (RANGER-4107) Upgrade EclipseLink
[
https://issues.apache.org/jira/browse/RANGER-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-4107:
---
Description:
See this comment from [~madhan] on RANGER-4104:
I was able to reproduce failures in execution of named query
{{XXAuthSession.getRecentAuthFailureCountByLoginId}} using a script that calls
Ranger REST APIs from multiple threads simultaneously with invalid credentials.
I notice at least 3 different types of failures:
1. ArrayOutOfBoundsException
{noformat}
[EL Warning]: 2023-02-22 20:36:40.8--UnitOfWork(729158064)--Exception
[EclipseLink-6168] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.QueryException
Exception Description: Query failed to prepare, unexpected error occurred:
[java.lang.ArrayIndexOutOfBoundsException: 2].
Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
Caused by: Exception [EclipseLink-6168] (Eclipse Persistence Services -
2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.QueryException
Exception Description: Query failed to prepare, unexpected error occurred:
[java.lang.ArrayIndexOutOfBoundsException: 2].
Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
Query: ReportQuery(name="XXAuthSession.getRecentAuthFailureCountByLoginId"
referenceClass=XXAuthSession sql="SELECT COUNT(?) FROM x_auth_sess t0 WHERE
(((t0.LOGIN_ID = ?) AND (t0.AUTH_STATUS <> ?)) AND (t0.CREATE_TIME >
COALESCE((SELECT MAX(t1.CREATE_TIME) FROM x_auth_sess t1 WHERE (((t1.LOGIN_ID =
?) AND (t1.AUTH_STATUS = ?)) AND (t1.CREATE_TIME > ?))),? )))")
at
org.eclipse.persistence.exceptions.QueryException.prepareFailed(QueryException.java:1590)
at
org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:680)
at
org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:901)
at
org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:613)
at
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:867)
at
org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
at
org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
at
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
at
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1751)
at
org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
at
org.eclipse.persistence.internal.jpa.QueryImpl.getSingleResult(QueryImpl.java:517)
... 66 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at
org.eclipse.persistence.expressions.ListExpressionOperator.getDatabaseStrings(ListExpressionOperator.java:75)
at
org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2196)
at
org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:99)
at
org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
at
org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:286)
at
org.eclipse.persistence.internal.expressions.RelationExpression.printSQL(RelationExpression.java:899)
at
org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2239)
at
org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:286)
at
org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.translateExpression(ExpressionSQLPrinter.java:306)
at
org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printExpression(ExpressionSQLPrinter.java:129)
at
org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL(SQLSelectStatement.java:1683)
at
org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.printSQLSelectStatement(DatabasePlatform.java:3178)
at
org.eclipse.persistence.platform.database.MySQLPlatform.printSQLSelectStatement(MySQLPlatform.java:673)
{noformat}
2. NullPointerException
{noformat}
[EL Warning]: 2023-02-22 20:36:40.8--UnitOfWork(305743338)--Exception
[EclipseLink-6168] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.QueryExcept
[jira] [Created] (RANGER-4107) Upgrade EclipseLink
Andrew Luo created RANGER-4107: -- Summary: Upgrade EclipseLink Key: RANGER-4107 URL: https://issues.apache.org/jira/browse/RANGER-4107 Project: Ranger Issue Type: Improvement Components: admin Reporter: Andrew Luo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692773#comment-17692773
]
Andrew Luo commented on RANGER-4104:
[~kirbyzhou] I commented on that issue.
I think there are multiple issues here. Upgrading EclipseLink will be useful
but it doesn't solve this problem. I can open a separate ticket/PR for that.
The specific problem I wanted to fix is that the java.util.Date object is
converted to a query string in the wrong format (this doesn't cause an error
per se but the feature does not work as it scans the wrong part of the table
and also does not use the index) and this causes MySQL (before 8.0.16). So in
older versions it will do a string comparison which cannot use indexes, in
8.0.17 it will fail (see https://bugs.mysql.com/bug.php?id=93513)
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692757#comment-17692757
]
Andrew Luo commented on RANGER-3756:
Also, upgrading EclipseLink as you suggested in RANGER-4104 does not help this
particular bug as it requires temp tables even in the latest 2.x version:
[eclipselink/ExpressionQueryMechanism.java at 2.7 · eclipse-ee4j/eclipselink ·
GitHub|https://github.com/eclipse-ee4j/eclipselink/blob/2.7/foundation/org.eclipse.persistence.core/src/org/eclipse/persistence/internal/queries/ExpressionQueryMechanism.java]
(Search for "// Main Case: Descriptor is mapped to more than one table and/or
the query references other tables"). Even in 3.x/master this is the case.
So there's only two solutions:
# Upgrade MySQL to >= 8.0.13
# Refactor the delete statement to get the IDs in the Java code and then do a
DELETE without any other table references (basically moving the temporary table
to Java side)
I'll let other people comment but I think upgrading MySQL is a reasonable
approach, since (2) would negatively affect performance on other
databases/non-GTID setups or otherwise add code complexity (if we were to only
write code for the < 8.0.13 GTID MySQL case)
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692728#comment-17692728
]
Andrew Luo commented on RANGER-3756:
[~kirbyzhou] the issue should only be reproducible on MySQL < 8.0.13 since
8.0.13+ adds support for temporary tables in GTID transactions: [MySQL ::
Temporary tables are now allowed in transactions when GTIDs are
enabled|https://dev.mysql.com/blog-archive/temporary-tables-are-now-allowed-in-transactions-when-gtids-are-enabled/]
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692267#comment-17692267
]
Andrew Luo commented on RANGER-4104:
The annotation on the entity class affects how the ResultSet from JDBC is
converted to an object, but when calling setParameter we still have to specify
TemporalType.DATE. Otherwise there would be no need for setParameter to take
that parameter if the entity column annotation was sufficient (from a design
perspective, I don't know exactly why this is but I suspect it's might to allow
compatible types to be compared, for example, Date and Time)
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692264#comment-17692264
]
Andrew Luo commented on RANGER-4104:
I didn't try reproducing on other databases but it is reproducible on MySQL.
If you login and look at the actual JDBC query generated in the log, you will
see that the format of the date is incorrect.
If you don't specify the TemporalType.DATE parameter, then it will use the
default parameter type conversion in JDBC 4.2 Appendix B.4 which specifies that
java.util.Date is mapped to TIMESTAMP type. If you use java.sql.Date then it's
mapped to DATE type. So either we should add the TemporalType.DATE parameter
or convert to java.sql.Date in the getRecentAuthFailureCountByLoginId method.
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691579#comment-17691579
]
Andrew Luo commented on RANGER-4104:
[https://reviews.apache.org/r/74313/]
https://github.com/apache/ranger/pull/220
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-4104:
---
Affects Version/s: 3.0.0
2.4.0
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-4104:
---
Attachment: 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-2362) [security] Admin webui - Lack of account lockout
[ https://issues.apache.org/jira/browse/RANGER-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691577#comment-17691577 ] Andrew Luo commented on RANGER-2362: The patch for this creates a bug: [RANGER-4104] XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code - ASF JIRA (apache.org) > [security] Admin webui - Lack of account lockout > > > Key: RANGER-2362 > URL: https://issues.apache.org/jira/browse/RANGER-2362 > Project: Ranger > Issue Type: Bug > Components: admin, Ranger >Affects Versions: 1.0.0 >Reporter: t oo >Assignee: kirby zhou >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > |Account lockout is a mechanism used to stop non-valid users from guessing > for the right password. It is also a protection against brute force attacks > wherein an automated system can use common/dictionary passwords or even build > passwords based on set of characters just to try to guess the valid one.| > |The application does not implement an account lockout mechanism, leaving it > susceptible to brute force attacks. These login pages were susceptible to > this condition.| > |It is possible for an attacker to use dictionary or brute force attacks and > set it to attempt sending the requests on a particular amount of time to > bypass the validation. Once a username has been correctly guessed, the > attacker may then be able to gain access to the application. Since it is > vulnerable to Form Auto Complete Active vulnerability (LINK) which makes the > email addresses easier to guess, it will make brute force attack to more > likely possible. > |Enforce account lockout conditions to prevent intrusions and improve > password requirements and complexities to avoid the chances of brute force > and dictionary attacks from working.| > | -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74313: RANGER-4104: XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74313/
---
Review request for ranger and Madhan Neethiraj.
Bugs: RANGER-4104
https://issues.apache.org/jira/browse/RANGER-4104
Repository: ranger
Description
---
XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
due to an error in how the authWindowStartTime Date parameter is bound.
It is currently bound with setParameter("authWindowStartTime",
authWindowStartTime) however, JPA 2.2 Specification Section 11.1.53 says that
it should be bound by specifying an additional parameter TemporalType.DATE.
This causes issues on some database dialects (at least MySQL) as it serializes
the date string incorrectly which scans the wrong set of data (causing
performance issues since the index cannot be used), and also the functionality
introduced in RANGER-2362 does not work properly
Diffs
-
security-admin/src/main/java/org/apache/ranger/db/XXAuthSessionDao.java
934d25865
Diff: https://reviews.apache.org/r/74313/diff/1/
Testing
---
Thanks,
Andrew Luo
[jira] [Created] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
Andrew Luo created RANGER-4104:
--
Summary: XXAuthSessionDao.getRecentAuthFailureCountByLoginId
produces incorrect SQL code
Key: RANGER-4104
URL: https://issues.apache.org/jira/browse/RANGER-4104
Project: Ranger
Issue Type: Bug
Components: admin
Reporter: Andrew Luo
XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
due to an error in how the authWindowStartTime Date parameter is bound.
It is currently bound with setParameter("authWindowStartTime",
authWindowStartTime) however, [JPA 2.2 Specification Section
11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
says that it should be bound by specifying an additional parameter
TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: (was: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch) > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > Time Spent: 10m > Remaining Estimate: 0h > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > Time Spent: 10m > Remaining Estimate: 0h > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/ --- (Updated Feb. 8, 2023, 9:05 p.m.) Review request for ranger and Pradeep Agrawal. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). We previously tried this commit and it caused a few issues (some typos on SQL Server and SQL Anywhere, as well as upgrade scenarios - however this has been fixed). Diffs (updated) - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 9a79fe8ad security-admin/db/mysql/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql fd6cec9a7 security-admin/db/oracle/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 4d5a8cedf security-admin/db/postgres/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 3ed2a5b9c security-admin/db/sqlanywhere/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql ca8f7da1f security-admin/db/sqlserver/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 6cc3509d8 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 691ab52b3 Diff: https://reviews.apache.org/r/74301/diff/3/ Changes: https://reviews.apache.org/r/74301/diff/2-3/ Testing --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Also searched the source code and JPA queries xml for other references to these fields Thanks, Andrew Luo
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: > 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: (was: 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch) > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: > 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74302: RANGER-4076: Support Java 17 for build and runtime
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74302/ --- (Updated Feb. 8, 2023, 8:46 p.m.) Review request for ranger. Bugs: RANGER-4076 https://issues.apache.org/jira/browse/RANGER-4076 Repository: ranger Description --- This requires upgrading a few maven plugins, libraries, as well as changing from Nashorn to GraalJS. This uses the latest version of GraalJS that supports JDK 8. If preferred, I can split this into smaller commits. Diffs (updated) - .mvn/jvm.config PRE-CREATION agents-common/pom.xml b753c1368 agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java db620df92 agents-common/src/main/java/org/apache/ranger/plugin/util/ScriptEngineUtil.java 8d76c1d81 hdfs-agent/pom.xml a23cdfea0 kms/pom.xml e97b993d7 kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 plugin-nestedstructure/pom.xml 0e208f99c plugin-nestedstructure/src/main/java/org/apache/ranger/authorization/nestedstructure/authorizer/RecordFilterJavaScript.java 77767767c plugin-nestedstructure/src/test/java/org/apache/ranger/authorization/nestedstructure/authorizer/TestRecordFilterJavaScript.java 9cb161b8d pom.xml e402bcc5d security-admin/pom.xml 2e5e21950 Diff: https://reviews.apache.org/r/74302/diff/4/ Changes: https://reviews.apache.org/r/74302/diff/3-4/ Testing --- Tested compile and run on both Java 8 and 17. File Attachments 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch https://reviews.apache.org/media/uploaded/files/2023/02/08/d87b4c7c-e84c-4ddb-b059-51c98bf75772__0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch Thanks, Andrew Luo
Re: Review Request 74302: RANGER-4076: Support Java 17 for build and runtime
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74302/ --- (Updated Feb. 8, 2023, 8:43 p.m.) Review request for ranger. Bugs: RANGER-4076 https://issues.apache.org/jira/browse/RANGER-4076 Repository: ranger Description --- This requires upgrading a few maven plugins, libraries, as well as changing from Nashorn to GraalJS. This uses the latest version of GraalJS that supports JDK 8. If preferred, I can split this into smaller commits. Diffs (updated) - .mvn/jvm.config PRE-CREATION agents-common/pom.xml b753c1368 agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java db620df92 agents-common/src/main/java/org/apache/ranger/plugin/util/ScriptEngineUtil.java 8d76c1d81 hdfs-agent/pom.xml a23cdfea0 kms/pom.xml e97b993d7 kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 plugin-nestedstructure/pom.xml 0e208f99c plugin-nestedstructure/src/main/java/org/apache/ranger/authorization/nestedstructure/authorizer/RecordFilterJavaScript.java 77767767c plugin-nestedstructure/src/test/java/org/apache/ranger/authorization/nestedstructure/authorizer/TestRecordFilterJavaScript.java 9cb161b8d pom.xml e402bcc5d security-admin/pom.xml 2e5e21950 Diff: https://reviews.apache.org/r/74302/diff/3/ Changes: https://reviews.apache.org/r/74302/diff/2-3/ Testing --- Tested compile and run on both Java 8 and 17. File Attachments (updated) 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch https://reviews.apache.org/media/uploaded/files/2023/02/08/d87b4c7c-e84c-4ddb-b059-51c98bf75772__0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch Thanks, Andrew Luo
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: > 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: (was: 0001-JDK-17.patch) > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: > 0001-RANGER-4076-Support-Java-17-for-build-and-runtime.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On Feb. 8, 2023, 4:54 p.m., Madhan Neethiraj wrote: > > @Andrew - thank you for this patch to improve performance of policy > > changes. Changes in the patch look good. > > > > I have few suggestions: > > > > 1. Column named guid in ref tables is not mapped to any field in respective > > Java entity class. It might help to remove this unused column. > > 2. DB patch to drop constriants look good. However, removing columns from > > tables would require following sequence of steps - to avoid earlier version > > Ranger running into errors while referring to these columns: > > - stop Ranger server instances > > - apply the DB patch to remove columns > > - update Ranger admin server to newer version containing this patch > > - start Ranger server instances > > > > This will result in Ranger being not available for the duration of above > > steps. This will not be acceptable in deployments having Ranger in high > > availablity configuration. > > > > I suggest not to remove columns that are used by current version of Ranger. > > Note that these columns can be removed from scripts that create Ranger > > schema in a new deployment (scripts under directory named optimized). > > Andrew Luo wrote: > I see. What I could recommend here is that we can not remove the columns > during the upgrade, and revisit this a few months/years later when we're > comfortable. > > Madhan Neethiraj wrote: > yes, that sounds good. These columns can be removed in next major > release, 3.0 i.e., master branch. > > Assuming this patch makes to 2.4 release, upgrade from 2.4 to 3.0 will > not require restart of Ranger - since Ranger 2.4 will not be referring to > columns deleted in 3.0. Okay. I'll comment out the SQL scripts - but I think I need to reserve that patch number so that when we eventually put it in 3.0, we have an indication that the "optimized" schema already has this patch. Thanks for the suggestion! - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/#review225166 --- On Feb. 8, 2023, 4:09 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74301/ > --- > > (Updated Feb. 8, 2023, 4:09 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > We previously tried this commit and it caused a few issues (some typos on SQL > Server and SQL Anywhere, as well as upgrade scenarios - however this has been > fixed). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 9a79fe8ad > > security-admin/db/mysql/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > fd6cec9a7 > > security-admin/db/oracle/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 4d5a8cedf > > security-admin/db/postgres/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 3ed2a5b9c > > security-admin/db/sqlanywhere/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > ca8f7da1f > > security-admin/db/sqlserver/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 6cc3509d8 > > security-admin/src/main/java/org/apache/ranger/entit
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On Feb. 8, 2023, 4:54 p.m., Madhan Neethiraj wrote: > > @Andrew - thank you for this patch to improve performance of policy > > changes. Changes in the patch look good. > > > > I have few suggestions: > > > > 1. Column named guid in ref tables is not mapped to any field in respective > > Java entity class. It might help to remove this unused column. > > 2. DB patch to drop constriants look good. However, removing columns from > > tables would require following sequence of steps - to avoid earlier version > > Ranger running into errors while referring to these columns: > > - stop Ranger server instances > > - apply the DB patch to remove columns > > - update Ranger admin server to newer version containing this patch > > - start Ranger server instances > > > > This will result in Ranger being not available for the duration of above > > steps. This will not be acceptable in deployments having Ranger in high > > availablity configuration. > > > > I suggest not to remove columns that are used by current version of Ranger. > > Note that these columns can be removed from scripts that create Ranger > > schema in a new deployment (scripts under directory named optimized). I see. What I could recommend here is that we can not remove the columns during the upgrade, and revisit this a few months/years later when we're comfortable. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/#review225166 --- On Feb. 8, 2023, 4:09 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74301/ > --- > > (Updated Feb. 8, 2023, 4:09 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > We previously tried this commit and it caused a few issues (some typos on SQL > Server and SQL Anywhere, as well as upgrade scenarios - however this has been > fixed). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 9a79fe8ad > > security-admin/db/mysql/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > fd6cec9a7 > > security-admin/db/oracle/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 4d5a8cedf > > security-admin/db/postgres/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > 3ed2a5b9c > > security-admin/db/sqlanywhere/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > ca8f7da1f > > security-admin/db/sqlserver/patches/061-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 6cc3509d8 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity
Re: Review Request 74245: RANGER-3899:Policy creation takes more time when there are more users, groups, roles
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74245/#review225167 --- I'm not a committer but I was the author of the original PR a few years back - happy to see this come back and hope it can be merged soon! - Andrew Luo On Jan. 4, 2023, 7:14 a.m., Ramachandran Krishnan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74245/ > --- > > (Updated Jan. 4, 2023, 7:14 a.m.) > > > Review request for ranger, Don Bosco Durai, Kirby Zhou, Abhay Kulkarni, > Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, > Selvamohan Neethiraj, Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan > Periasamy. > > > Bugs: RANGER-3899 > https://issues.apache.org/jira/browse/RANGER-3899 > > > Repository: ranger > > > Description > --- > > Analysis is when policy has 500 users with 5 policy items. > > we perform total 526 DB read calls and 504 DB write, 3 DB Batch write calls. > This is the reason for policy creation time where 1000 plus DB calls are done > sequentially. > Time is taken to create the policy :4~8 seconds > > 1st#Improvement: > > After Fix :After changing to batch create of XXPolicyRefUser instead of > calling 1 DB write XXPolicyRefUser for every user of the policy: > > When Policy contains 500 users,5 access Types (permissions),12 resources > (1DB,1Table,10 columns) > > we perform total 526 DB read calls and 6 DB write calls, 5 DB Batch write > calls. > > Time is taken to create the policy :2.6~4 seconds > > 2nd#Improvement: > Getting the ID for each role/group/user one-by-one is slow, especially for > large policies with many roles/groups/users. Batching significantly improves > performance. > > Fixing this part will reduce the DB read calls to a very minimum which will > improve performance drastically. > > The policy contains 500 users,5 access Types (permissions),12 resources > (1DB,1Table,10 columns) > > After fix: > DB Read count:27 > > Time taken to create the policy: 1.4 to 2 secs (The policy contains 500 > users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)) > > DB read calls will be reduced to 27 from 526 > > Time is taken to create the policy :1.4~2 seconds > > > Diffs > - > > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 6cc3509d8 > security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 8ca4f62bb > security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 4e5b692b5 > security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 97bc2680a > security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213 > > > Diff: https://reviews.apache.org/r/74245/diff/2/ > > > Testing > --- > > Tested Created policies via load testing > > > Thanks, > > Ramachandran Krishnan > >
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: (was: 0001-JDK-17.patch) > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-JDK-17.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74302: RANGER-4076: Support Java 17 for build and runtime
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74302/ --- (Updated Feb. 8, 2023, 4:49 p.m.) Review request for ranger. Bugs: RANGER-4076 https://issues.apache.org/jira/browse/RANGER-4076 Repository: ranger Description --- This requires upgrading a few maven plugins, libraries, as well as changing from Nashorn to GraalJS. This uses the latest version of GraalJS that supports JDK 8. If preferred, I can split this into smaller commits. Diffs (updated) - .mvn/jvm.config PRE-CREATION agents-common/pom.xml b753c1368 agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java db620df92 agents-common/src/main/java/org/apache/ranger/plugin/util/ScriptEngineUtil.java 8d76c1d81 hdfs-agent/pom.xml a23cdfea0 kms/pom.xml e97b993d7 kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 plugin-nestedstructure/pom.xml 0e208f99c plugin-nestedstructure/src/main/java/org/apache/ranger/authorization/nestedstructure/authorizer/RecordFilterJavaScript.java 77767767c plugin-nestedstructure/src/test/java/org/apache/ranger/authorization/nestedstructure/authorizer/TestRecordFilterJavaScript.java 9cb161b8d pom.xml e402bcc5d security-admin/pom.xml 2e5e21950 Diff: https://reviews.apache.org/r/74302/diff/2/ Changes: https://reviews.apache.org/r/74302/diff/1-2/ Testing --- Tested compile and run on both Java 8 and 17. Thanks, Andrew Luo
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: (was: 0001-JDK-17.patch) > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-JDK-17.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-JDK-17.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: (was: 0001-JDK-17.patch) > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-4076: --- Attachment: 0001-JDK-17.patch > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4076) Support Java 17 for build and runtime
[ https://issues.apache.org/jira/browse/RANGER-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685979#comment-17685979 ] Andrew Luo commented on RANGER-4076: [https://reviews.apache.org/r/74302/] [https://github.com/apache/ranger/pull/212] > Support Java 17 for build and runtime > - > > Key: RANGER-4076 > URL: https://issues.apache.org/jira/browse/RANGER-4076 > Project: Ranger > Issue Type: New Feature > Components: admin, build-infra > Reporter: Andrew Luo >Priority: Major > Attachments: 0001-JDK-17.patch > > > Currently only Java 8 and 11 are supported. Java 17 is a major LTS version > of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74302: RANGER-4076: Support Java 17 for build and runtime
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74302/ --- (Updated Feb. 8, 2023, 4:46 p.m.) Review request for ranger. Bugs: RANGER-4076 https://issues.apache.org/jira/browse/RANGER-4076 Repository: ranger Description (updated) --- This requires upgrading a few maven plugins, libraries, as well as changing from Nashorn to GraalJS. This uses the latest version of GraalJS that supports JDK 8. If preferred, I can split this into smaller commits. Diffs - .mvn/jvm.config PRE-CREATION agents-common/pom.xml b753c1368 agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java db620df92 agents-common/src/main/java/org/apache/ranger/plugin/util/ScriptEngineUtil.java 8d76c1d81 hdfs-agent/pom.xml a23cdfea0 kms/pom.xml e97b993d7 kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 plugin-nestedstructure/pom.xml 0e208f99c plugin-nestedstructure/src/main/java/org/apache/ranger/authorization/nestedstructure/authorizer/RecordFilterJavaScript.java 77767767c plugin-nestedstructure/src/test/java/org/apache/ranger/authorization/nestedstructure/authorizer/TestRecordFilterJavaScript.java 9cb161b8d pom.xml e402bcc5d ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/TrinoRangerPlugin.java aa6ce8620 security-admin/pom.xml 2e5e21950 Diff: https://reviews.apache.org/r/74302/diff/1/ Testing --- Tested compile and run on both Java 8 and 17. Thanks, Andrew Luo
Review Request 74302: RANGER-4076: Support Java 17 for build and runtime
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74302/ --- Review request for ranger. Bugs: RANGER-4076 https://issues.apache.org/jira/browse/RANGER-4076 Repository: ranger Description --- This requires upgrading a few maven plugins, libraries, as well as changing from Nashorn to GraalJS. This uses the latest version of GraalJS that supports JDK 8. Diffs - .mvn/jvm.config PRE-CREATION agents-common/pom.xml b753c1368 agents-common/src/main/java/org/apache/ranger/plugin/util/NashornScriptEngineCreator.java db620df92 agents-common/src/main/java/org/apache/ranger/plugin/util/ScriptEngineUtil.java 8d76c1d81 hdfs-agent/pom.xml a23cdfea0 kms/pom.xml e97b993d7 kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 plugin-nestedstructure/pom.xml 0e208f99c plugin-nestedstructure/src/main/java/org/apache/ranger/authorization/nestedstructure/authorizer/RecordFilterJavaScript.java 77767767c plugin-nestedstructure/src/test/java/org/apache/ranger/authorization/nestedstructure/authorizer/TestRecordFilterJavaScript.java 9cb161b8d pom.xml e402bcc5d ranger-trino-plugin-shim/src/main/java/org/apache/ranger/authorization/trino/authorizer/TrinoRangerPlugin.java aa6ce8620 security-admin/pom.xml 2e5e21950 Diff: https://reviews.apache.org/r/74302/diff/1/ Testing --- Tested compile and run on both Java 8 and 17. Thanks, Andrew Luo
[jira] [Created] (RANGER-4076) Support Java 17 for build and runtime
Andrew Luo created RANGER-4076: -- Summary: Support Java 17 for build and runtime Key: RANGER-4076 URL: https://issues.apache.org/jira/browse/RANGER-4076 Project: Ranger Issue Type: New Feature Components: admin, build-infra Reporter: Andrew Luo Currently only Java 8 and 11 are supported. Java 17 is a major LTS version of Java and adding support would modernize our Java version support. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685975#comment-17685975 ] Andrew Luo commented on RANGER-2713: https://github.com/apache/ranger/pull/211 > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > Time Spent: 10m > Remaining Estimate: 0h > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685971#comment-17685971 ] Andrew Luo commented on RANGER-2713: https://reviews.apache.org/r/74301/ > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: (was: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch) > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: (was: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch) > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/ --- (Updated Feb. 8, 2023, 4:09 p.m.) Review request for ranger and Pradeep Agrawal. Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). We previously tried this commit and it caused a few issues (some typos on SQL Server and SQL Anywhere, as well as upgrade scenarios - however this has been fixed). Diffs (updated) - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 9a79fe8ad security-admin/db/mysql/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql fd6cec9a7 security-admin/db/oracle/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 4d5a8cedf security-admin/db/postgres/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 3ed2a5b9c security-admin/db/sqlanywhere/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql ca8f7da1f security-admin/db/sqlserver/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 6cc3509d8 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 691ab52b3 Diff: https://reviews.apache.org/r/74301/diff/2/ Changes: https://reviews.apache.org/r/74301/diff/1-2/ Testing --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Also searched the source code and JPA queries xml for other references to these fields Thanks, Andrew Luo
Review Request 74301: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74301/ --- Review request for ranger and Pradeep Agrawal. Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). We previously tried this commit and it caused a few issues (some typos on SQL Server and SQL Anywhere, as well as upgrade scenarios - however this has been fixed). Diffs - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 9a79fe8ad security-admin/db/mysql/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql fd6cec9a7 security-admin/db/oracle/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 4d5a8cedf security-admin/db/postgres/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 3ed2a5b9c security-admin/db/sqlanywhere/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql ca8f7da1f security-admin/db/sqlserver/patches/061-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 6cc3509d8 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 691ab52b3 Diff: https://reviews.apache.org/r/74301/diff/1/ Testing --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Also searched the source code and JPA queries xml for other references to these fields Thanks, Andrew Luo
[jira] [Reopened] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo reopened RANGER-2713: > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch, > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Review Request 72205: RANGER-2752: Add corresponding upgrade patch for RANGER-2550
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72205/#review221003
---
Seems to be replaced by https://reviews.apache.org/r/72383/
- Andrew Luo
On March 6, 2020, 4:28 a.m., Andrew Luo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72205/
> ---
>
> (Updated March 6, 2020, 4:28 a.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2752
> https://issues.apache.org/jira/browse/RANGER-2752
>
>
> Repository: ranger
>
>
> Description
> ---
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verific
[jira] [Commented] (RANGER-2752) Upgrade from 0.7.0 to master fails due to schema issue
[
https://issues.apache.org/jira/browse/RANGER-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055669#comment-17055669
]
Andrew Luo commented on RANGER-2752:
[~pradeep] just wanted to get your thoughts on this. If you think we don't
need this patch I can close the review and this JIRA.
> Upgrade from 0.7.0 to master fails due to schema issue
> --
>
> Key: RANGER-2752
> URL: https://issues.apache.org/jira/browse/RANGER-2752
> Project: Ranger
> Issue Type: Bug
> Components: admin, Ranger
>Affects Versions: 0.7.0
> Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-RANGER-2752-Add-corresponding-upgrade-patch-for-RANGER-2550.patch,
> ranger_admin_sql_db_patch.log
>
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL
[jira] [Commented] (RANGER-2752) Upgrade from 0.7.0 to master fails due to schema issue
[
https://issues.apache.org/jira/browse/RANGER-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054193#comment-17054193
]
Andrew Luo commented on RANGER-2752:
The upgrade will fail every time for anyone upgrading without these DB schema
change patches, and policies won't get migrated since the subsequent Java
patches won't apply if a preceding one fails (one of the failing patches,
J10015, precedes J10019 which is turns the old policy schema into the new JSON
string schema).
My feeling is that if everyone has to apply the patch manually even if it takes
a long time, it would be better for the upgrade to do it automatically.
Besides, these tables are generally much smaller than the
x_policy_item_user_perm table, which was already migrated previously:
[https://github.com/apache/ranger/blob/master/security-admin/db/mysql/patches/024-sortorder-column-size.sql]
That said, it's up to you to decide, but I can change the patch to update only
the table x_access_type_def and it would probably fix this specific issue. But
longer term this could become problematic as the upgrade schema may be
different from new installs.
> Upgrade from 0.7.0 to master fails due to schema issue
> --
>
> Key: RANGER-2752
> URL: https://issues.apache.org/jira/browse/RANGER-2752
> Project: Ranger
> Issue Type: Bug
> Components: admin, Ranger
>Affects Versions: 0.7.0
> Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-RANGER-2752-Add-corresponding-upgrade-patch-for-RANGER-2550.patch,
> ranger_admin_sql_db_patch.log
>
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ require
Re: Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote: > > From which version upgrade case has been tested ? can you try from > > ranger-0.7 also if it was not tested? > > Andrew Luo wrote: > I tested from the previous build (without my change) to a version with my > change. I'll also test from 0.7 and let you know the results. > > Andrew Luo wrote: > Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar > test: > > 1. Install 0.7.0 > 2. Created new user > 3. Logged in as new user, created a policy > 4. Upgraded to my local build > 5. Logged back in as original user, delete user that created policy > 6. Verified no errors, policy is still present/correct > > I found some other bugs during the upgrade with the patches related to > Kafka (J10015, J10025, J10033) but the error was showing it was failing to > insert into x_access_type_def. I didn't change that table in my patch so I > think its a different issue (I will file a bug later if I repro this on a > build without my changes). Anyways, during the upgrade I just skipped those > patches by manually inserting those records into the table so the upgrade > script would think those patches already got applied, and everything worked > as expected. > > Andrew Luo wrote: > (I also tested policy creation after upgrade) > > Pradeep Agrawal wrote: > Thanks Andrew. I will also see what are the issues while upgrading from > ranger-0.7 to ranger-2.1 I found the issue: https://reviews.apache.org/r/72205 - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/#review219731 --- On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72166/ > --- > > (Updated Feb. 26, 2020, 10:18 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 47618f6b3 > > security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > e59e7de61 > > security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 37ea61912 > > security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > bd1c47cc4 > > security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > 22e1746f2 > > security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > baacfa4d8 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8df
Re: Review Request 72204: RANGER-2752: Add corresponding upgrade patch for RANGER-2550
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72204/#review219811
---
Oops, forgot to add the patch. Updated review is here:
https://reviews.apache.org/r/72205/
- Andrew Luo
On March 6, 2020, 4:27 a.m., Andrew Luo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72204/
> ---
>
> (Updated March 6, 2020, 4:27 a.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2752
> None
>
>
> Description
> ---
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> [EL Warning]: 2020-03-04 07:38:19.282-ClientSession(841
Review Request 72205: RANGER-2752: Add corresponding upgrade patch for RANGER-2550
/opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c \; -query "delete from x_db_version_h where version = 'J10015' and active = 'N' and updated_by='306a9688fa98';" Wed Mar 04 07:38:19 UTC 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2020-03-04 07:38:20,091 [E] applying java patch PatchForKafkaServiceDefUpdate_J10015 failed This seems to be due to this commit not having corresponding handling (DDL patch) for upgrade scenarios: https://github.com/apache/ranger/commit/2a6dd3d078c52cc1314af581f7b8e6c8cdab933a Diffs - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3 security-admin/db/mysql/patches/048-increase-sort-order.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61 security-admin/db/oracle/patches/048-increase-sort-order.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912 security-admin/db/postgres/patches/048-increase-sort-order.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4 security-admin/db/sqlanywhere/patches/048-increase-sort-order.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2 security-admin/db/sqlserver/patches/048-increase-sort-order.sql PRE-CREATION Diff: https://reviews.apache.org/r/72205/diff/1/ Testing --- Tested 0.7.0 to 2.1.0 upgrade on MySQL. Unfortunately I don't have 0.7.0 environments for upgrade testing on other DBs but I copied much of the patch syntax from patch 024, which changes the same column for other tables. Thanks, Andrew Luo
Review Request 72204: RANGER-2752: Add corresponding upgrade patch for RANGER-2550
er-2.1.0-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c \; -query "delete from x_db_version_h where version = 'J10015' and active = 'N' and updated_by='306a9688fa98';" Wed Mar 04 07:38:19 UTC 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2020-03-04 07:38:20,091 [E] applying java patch PatchForKafkaServiceDefUpdate_J10015 failed This seems to be due to this commit not having corresponding handling (DDL patch) for upgrade scenarios: https://github.com/apache/ranger/commit/2a6dd3d078c52cc1314af581f7b8e6c8cdab933a Testing --- Tested 0.7.0 to 2.1.0 upgrade on MySQL. Unfortunately I don't have 0.7.0 environments for upgrade testing on other DBs but I copied much of the patch syntax from patch 024, which changes the same column for other tables. Thanks, Andrew Luo
[jira] [Updated] (RANGER-2752) Upgrade from 0.7.0 to master fails due to schema issue
[
https://issues.apache.org/jira/browse/RANGER-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-2752:
---
Attachment:
0001-RANGER-2752-Add-corresponding-upgrade-patch-for-RANGER-2550.patch
> Upgrade from 0.7.0 to master fails due to schema issue
> --
>
> Key: RANGER-2752
> URL: https://issues.apache.org/jira/browse/RANGER-2752
> Project: Ranger
> Issue Type: Bug
> Components: admin, Ranger
>Affects Versions: 0.7.0
> Reporter: Andrew Luo
>Priority: Major
> Fix For: master
>
> Attachments:
> 0001-RANGER-2752-Add-corresponding-upgrade-patch-for-RANGER-2550.patch,
> ranger_admin_sql_db_patch.log
>
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate
[jira] [Created] (RANGER-2752) Upgrade from 0.7.0 to master fails due to schema issue
Andrew Luo created RANGER-2752:
--
Summary: Upgrade from 0.7.0 to master fails due to schema issue
Key: RANGER-2752
URL: https://issues.apache.org/jira/browse/RANGER-2752
Project: Ranger
Issue Type: Bug
Components: admin, Ranger
Affects Versions: 0.7.0
Reporter: Andrew Luo
Fix For: master
Attachments: ranger_admin_sql_db_patch.log
Upgrading from 0.7.0 to the current master fails with the following error:
2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
-cp
/opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
org.apache.util.sql.Jisql -driver mysqlconj -cstring
jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
\; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at,
updated_by,active) values ('J10015', current_timestamp, 'Ranger
2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015 is
being applied..
Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without server's
identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+
and 5.7.6+ requirements SSL connection must be established by default if
explicit option isn't set. For compliance with existing applications not using
SSL the verifyServerCertificate property is set to 'false'. You need either to
explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide
truststore for server certificate verification.
[EL Warning]: 2020-03-04 07:38:19.282--ClientSession(841450293)--Exception
[EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of
range value for column 'sort_order' at row 1
Error Code: 1264
Call: INSERT INTO x_access_type_def (ADDED_BY_ID, CREATE_TIME,
datamask_options, def_id, item_id, label, name, sort_order, rb_key_label,
rowfilter_options, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?)
bind => [12 parameters bound]
Query: ValueReadQuery(name="x_access_type_def_SEQ" sql="SELECT
L
[jira] [Updated] (RANGER-2752) Upgrade from 0.7.0 to master fails due to schema issue
[
https://issues.apache.org/jira/browse/RANGER-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-2752:
---
Attachment: ranger_admin_sql_db_patch.log
> Upgrade from 0.7.0 to master fails due to schema issue
> --
>
> Key: RANGER-2752
> URL: https://issues.apache.org/jira/browse/RANGER-2752
> Project: Ranger
> Issue Type: Bug
> Components: admin, Ranger
>Affects Versions: 0.7.0
> Reporter: Andrew Luo
>Priority: Major
> Fix For: master
>
> Attachments: ranger_admin_sql_db_patch.log
>
>
> Upgrading from 0.7.0 to the current master fails with the following error:
>
> 2020-03-04 07:38:00,523 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -cp
> /opt/mysql-connector-java-5.1.48.jar:/opt/ranger-2.1.0-SNAPSHOT-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://ranger-mysql/ranger -u 'ranger' -p '' -noheader -trim -c
> \; -query "insert into x_db_version_h (version, inst_at, inst_by,
> updated_at, updated_by,active) values ('J10015', current_timestamp, 'Ranger
> 2.1.0-SNAPSHOT', current_timestamp, '306a9688fa98','N') ;"
> Wed Mar 04 07:38:01 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> 2020-03-04 07:38:01,164 [I] java patch PatchForKafkaServiceDefUpdate_J10015
> is being applied..
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> Wed Mar 04 07:38:11 UTC 2020 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate verification.
> [EL Warning]: 2020-03-04 07:38:19.282--ClientSession(841450293)--Exception
> [EclipseLink-4002] (Eclipse Persis
Re: Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote: > > From which version upgrade case has been tested ? can you try from > > ranger-0.7 also if it was not tested? > > Andrew Luo wrote: > I tested from the previous build (without my change) to a version with my > change. I'll also test from 0.7 and let you know the results. > > Andrew Luo wrote: > Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar > test: > > 1. Install 0.7.0 > 2. Created new user > 3. Logged in as new user, created a policy > 4. Upgraded to my local build > 5. Logged back in as original user, delete user that created policy > 6. Verified no errors, policy is still present/correct > > I found some other bugs during the upgrade with the patches related to > Kafka (J10015, J10025, J10033) but the error was showing it was failing to > insert into x_access_type_def. I didn't change that table in my patch so I > think its a different issue (I will file a bug later if I repro this on a > build without my changes). Anyways, during the upgrade I just skipped those > patches by manually inserting those records into the table so the upgrade > script would think those patches already got applied, and everything worked > as expected. (I also tested policy creation after upgrade) - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/#review219731 ------- On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72166/ > --- > > (Updated Feb. 26, 2020, 10:18 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 47618f6b3 > > security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > e59e7de61 > > security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 37ea61912 > > security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > bd1c47cc4 > > security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > 22e1746f2 > > security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > baacfa4d8 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8dfb92833 > > security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java > 85e457efa > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > 69c8a4cb6 > > > Diff:
Re: Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote: > > From which version upgrade case has been tested ? can you try from > > ranger-0.7 also if it was not tested? > > Andrew Luo wrote: > I tested from the previous build (without my change) to a version with my > change. I'll also test from 0.7 and let you know the results. Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar test: 1. Install 0.7.0 2. Created new user 3. Logged in as new user, created a policy 4. Upgraded to my local build 5. Logged back in as original user, delete user that created policy 6. Verified no errors, policy is still present/correct I found some other bugs during the upgrade with the patches related to Kafka (J10015, J10025, J10033) but the error was showing it was failing to insert into x_access_type_def. I didn't change that table in my patch so I think its a different issue (I will file a bug later if I repro this on a build without my changes). Anyways, during the upgrade I just skipped those patches by manually inserting those records into the table so the upgrade script would think those patches already got applied, and everything worked as expected. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/#review219731 --- On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72166/ > --- > > (Updated Feb. 26, 2020, 10:18 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 47618f6b3 > > security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > e59e7de61 > > security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 37ea61912 > > security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > bd1c47cc4 > > security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > 22e1746f2 > > security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > baacfa4d8 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8dfb92833 > > security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java > 85e457efa > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > 69c8a4cb6 > > > Diff: https://reviews.apache.org/r/72166/diff/1/ > > > Testing > --- > > Fresh install and upgrade tested on all 5 databases (tested service and > policy creation) > > Also test
Re: Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote: > > From which version upgrade case has been tested ? can you try from > > ranger-0.7 also if it was not tested? I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/#review219731 --- On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72166/ > --- > > (Updated Feb. 26, 2020, 10:18 p.m.) > > > Review request for ranger and Pradeep Agrawal. > > > Bugs: RANGER-2713 > https://issues.apache.org/jira/browse/RANGER-2713 > > > Repository: ranger > > > Description > --- > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). > > > Diffs > - > > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql > 47618f6b3 > > security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql > e59e7de61 > > security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql > 37ea61912 > > security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > > security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql > bd1c47cc4 > > security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql > 22e1746f2 > > security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql > PRE-CREATION > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > baacfa4d8 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java > 6af8f99f4 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java > 4f4409d6a > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java > cb926740e > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java > 32a1b9f24 > > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java > 115064621 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java > 7aee502e0 > security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java > 8dfb92833 > > security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java > 85e457efa > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > 69c8a4cb6 > > > Diff: https://reviews.apache.org/r/72166/diff/1/ > > > Testing > --- > > Fresh install and upgrade tested on all 5 databases (tested service and > policy creation) > > Also tested user deletion on MySQL: > > 1. Created new user > 2. Logged in as new user, created a policy > 3. Logged back in as original user, delete user that created policy > 4. Verified no errors, policy is still present/correct > > Also searched the source code and JPA queries xml for other references to > these fields > > > Thanks, > > Andrew Luo > >
[jira] [Commented] (RANGER-2742) Database patch 046 does not apply correctly on SQL Server and SQL Anywhere
[ https://issues.apache.org/jira/browse/RANGER-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046145#comment-17046145 ] Andrew Luo commented on RANGER-2742: New review request, bundled together with the resubmit for RANGER-2713: https://reviews.apache.org/r/72166/ > Database patch 046 does not apply correctly on SQL Server and SQL Anywhere > -- > > Key: RANGER-2742 > URL: https://issues.apache.org/jira/browse/RANGER-2742 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master > Reporter: Andrew Luo >Priority: Critical > Fix For: master > > Attachments: > 0001-RANGER-2742-Fix-database-patch-046-on-SQL-Server-and-SQL-Anywher.patch > > > On SQL Anywhere the constraint and column names are swapped in some > statements. In SQL Server, the same problem exists in addition to not > properly dropping the default constraint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046144#comment-17046144 ] Andrew Luo commented on RANGER-2713: New review request: https://reviews.apache.org/r/72166/ > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Fix For: master > > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch, > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72164: RANGER-2742: Fix database patch 046 on SQL Server and SQL Anywhere
> On Feb. 27, 2020, 4:38 a.m., Pradeep Agrawal wrote: > > Can you consolidate these changes also with RANGER-2713 RR patch ? Sorry, I already did it: https://reviews.apache.org/r/72166/ I just forgot to close this RR. Closing now. Thanks! - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72164/#review219669 --- On Feb. 26, 2020, 5:41 a.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72164/ > --- > > (Updated Feb. 26, 2020, 5:41 a.m.) > > > Review request for ranger. > > > Bugs: RANGER-2742 > https://issues.apache.org/jira/browse/RANGER-2742 > > > Repository: ranger > > > Description > --- > > On SQL Anywhere the constraint and column names are swapped in some > statements. In SQL Server, the same problem exists in addition to not > properly dropping the default constraint. > > > Diffs > - > > > security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql > 10508f6ff > > security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql > 1af3ec449 > > > Diff: https://reviews.apache.org/r/72164/diff/1/ > > > Testing > --- > > Tested end-to-end on SQL Anywhere and SQL Server, created policies. > > > Thanks, > > Andrew Luo > >
Re: Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/ --- (Updated Feb. 26, 2020, 10:18 p.m.) Review request for ranger and Pradeep Agrawal. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). Diffs - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3 security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61 security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912 security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4 security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2 security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6 Diff: https://reviews.apache.org/r/72166/diff/1/ Testing (updated) --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Also searched the source code and JPA queries xml for other references to these fields Thanks, Andrew Luo
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Fix For: master > > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch, > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72166: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72166/ --- Review request for ranger and Pradeep Agrawal. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). Diffs - security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3 security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61 security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912 security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4 security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2 security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6 Diff: https://reviews.apache.org/r/72166/diff/1/ Testing --- Fresh install and upgrade tested on all 5 databases (tested service and policy creation) Also tested user deletion on MySQL: 1. Created new user 2. Logged in as new user, created a policy 3. Logged back in as original user, delete user that created policy 4. Verified no errors, policy is still present/correct Thanks, Andrew Luo
Review Request 72164: RANGER-2742: Fix database patch 046 on SQL Server and SQL Anywhere
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72164/ --- Review request for ranger. Bugs: RANGER-2742 https://issues.apache.org/jira/browse/RANGER-2742 Repository: ranger Description --- On SQL Anywhere the constraint and column names are swapped in some statements. In SQL Server, the same problem exists in addition to not properly dropping the default constraint. Diffs - security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql 10508f6ff security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql 1af3ec449 Diff: https://reviews.apache.org/r/72164/diff/1/ Testing --- Tested end-to-end on SQL Anywhere and SQL Server, created policies. Thanks, Andrew Luo
[jira] [Updated] (RANGER-2742) Database patch 046 does not apply correctly on SQL Server and SQL Anywhere
[ https://issues.apache.org/jira/browse/RANGER-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2742: --- Attachment: 0001-RANGER-2742-Fix-database-patch-046-on-SQL-Server-and-SQL-Anywher.patch > Database patch 046 does not apply correctly on SQL Server and SQL Anywhere > -- > > Key: RANGER-2742 > URL: https://issues.apache.org/jira/browse/RANGER-2742 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master > Reporter: Andrew Luo >Priority: Critical > Fix For: master > > Attachments: > 0001-RANGER-2742-Fix-database-patch-046-on-SQL-Server-and-SQL-Anywher.patch > > > On SQL Anywhere the constraint and column names are swapped in some > statements. In SQL Server, the same problem exists in addition to not > properly dropping the default constraint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2742) Database patch 046 does not apply correctly on SQL Server and SQL Anywhere
Andrew Luo created RANGER-2742: -- Summary: Database patch 046 does not apply correctly on SQL Server and SQL Anywhere Key: RANGER-2742 URL: https://issues.apache.org/jira/browse/RANGER-2742 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: master Reporter: Andrew Luo Fix For: master On SQL Anywhere the constraint and column names are swapped in some statements. In SQL Server, the same problem exists in addition to not properly dropping the default constraint. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72142: RANGER-2732: Batch lookup role, group, and user IDs during policy creation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72142/ --- (Updated Feb. 17, 2020, 7:56 p.m.) Review request for ranger. Changes --- Addressed review comments Bugs: RANGER-2732 https://issues.apache.org/jira/browse/RANGER-2732 Repository: ranger Description --- Getting the ID for each role/group/user one-by-one is slow, especially for large policies with many roles/groups/users. Batching significantly improves performance. Diffs (updated) - security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 318f9f505 security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2 security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 8528652fc security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java cea90c165 security-admin/src/main/resources/META-INF/jpa_named_queries.xml f23bf2e65 Diff: https://reviews.apache.org/r/72142/diff/2/ Changes: https://reviews.apache.org/r/72142/diff/1-2/ Testing --- Tested end-to-end. Verified that missing users are created, and existing users are correctly added to policies. Thanks, Andrew Luo
Re: Review Request 72142: RANGER-2732: Batch lookup role, group, and user IDs during policy creation
> On Feb. 17, 2020, 1:31 a.m., Pradeep Agrawal wrote: > > security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java > > Lines 76 (patched) > > <https://reviews.apache.org/r/72142/diff/1/?file=2211449#file2211449line77> > > > > Try to use CollectionUtils.isEmpty(groupNames) here from package > > org.apache.commons.collections; > > > > update the same in other places also. Thanks, will fix. > On Feb. 17, 2020, 1:31 a.m., Pradeep Agrawal wrote: > > security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java > > Lines 77 (patched) > > <https://reviews.apache.org/r/72142/diff/1/?file=2211449#file2211449line78> > > > > if possible avoid multiple return statements from the same method. I just followed the structure of the existing findByGroupName/findByRoleName/findByUserName functions. I can change it if you believe this is the right way, let me know what you prefer. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72142/#review219598 ------- On Feb. 16, 2020, 10:48 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72142/ > --- > > (Updated Feb. 16, 2020, 10:48 p.m.) > > > Review request for ranger. > > > Bugs: RANGER-2732 > https://issues.apache.org/jira/browse/RANGER-2732 > > > Repository: ranger > > > Description > --- > > Getting the ID for each role/group/user one-by-one is slow, especially for > large policies with many roles/groups/users. Batching significantly improves > performance. > > > Diffs > - > > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 318f9f505 > security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2 > security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 8528652fc > security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java cea90c165 > security-admin/src/main/resources/META-INF/jpa_named_queries.xml f23bf2e65 > > > Diff: https://reviews.apache.org/r/72142/diff/1/ > > > Testing > --- > > Tested end-to-end. Verified that missing users are created, and existing > users are correctly added to policies. > > > Thanks, > > Andrew Luo > >
Re: Review Request 72142: RANGER-2732: Batch lookup role, group, and user IDs during policy creation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72142/ --- (Updated Feb. 16, 2020, 10:48 p.m.) Review request for ranger. Summary (updated) - RANGER-2732: Batch lookup role, group, and user IDs during policy creation Bugs: RANGER-2732 https://issues.apache.org/jira/browse/RANGER-2732 Repository: ranger Description --- Getting the ID for each role/group/user one-by-one is slow, especially for large policies with many roles/groups/users. Batching significantly improves performance. Diffs - security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 318f9f505 security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2 security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 8528652fc security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java cea90c165 security-admin/src/main/resources/META-INF/jpa_named_queries.xml f23bf2e65 Diff: https://reviews.apache.org/r/72142/diff/1/ Testing --- Tested end-to-end. Verified that missing users are created, and existing users are correctly added to policies. Thanks, Andrew Luo
[jira] [Updated] (RANGER-2732) Batch lookup role, group, and user IDs during policy creation
[ https://issues.apache.org/jira/browse/RANGER-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2732: --- Attachment: 0001-RANGER-2732-Batch-lookup-role-group-and-user-IDs-during-policy-c.patch > Batch lookup role, group, and user IDs during policy creation > - > > Key: RANGER-2732 > URL: https://issues.apache.org/jira/browse/RANGER-2732 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: master > Reporter: Andrew Luo >Priority: Major > Fix For: master > > Attachments: > 0001-RANGER-2732-Batch-lookup-role-group-and-user-IDs-during-policy-c.patch > > > Getting the ID for each role/group/user one-by-one is slow, especially for > large policies with many roles/groups/users. Batching significantly improves > performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72142: RANGER-2732: Batch lookup role, group, and user IDs during policy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72142/ --- Review request for ranger. Bugs: RANGER-2732 https://issues.apache.org/jira/browse/RANGER-2732 Repository: ranger Description --- Getting the ID for each role/group/user one-by-one is slow, especially for large policies with many roles/groups/users. Batching significantly improves performance. Diffs - security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 318f9f505 security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2 security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 8528652fc security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java cea90c165 security-admin/src/main/resources/META-INF/jpa_named_queries.xml f23bf2e65 Diff: https://reviews.apache.org/r/72142/diff/1/ Testing --- Tested end-to-end. Verified that missing users are created, and existing users are correctly added to policies. Thanks, Andrew Luo
[jira] [Created] (RANGER-2732) Batch lookup role, group, and user IDs during policy creation
Andrew Luo created RANGER-2732: -- Summary: Batch lookup role, group, and user IDs during policy creation Key: RANGER-2732 URL: https://issues.apache.org/jira/browse/RANGER-2732 Project: Ranger Issue Type: Improvement Components: admin Affects Versions: master Reporter: Andrew Luo Fix For: master Getting the ID for each role/group/user one-by-one is slow, especially for large policies with many roles/groups/users. Batching significantly improves performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72140: RANGER-2731: Use correct property for batch size when batch persisting
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72140/ --- Review request for ranger. Bugs: RANGER-2731 https://issues.apache.org/jira/browse/RANGER-2731 Repository: ranger Description --- BaseDao.batchCreate uses the property ranger.jpa.jdbc.batch-clear.size instead of the correct property, ranger.jpa.jdbc.batch-persist.size Diffs - security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 98837fe38 Diff: https://reviews.apache.org/r/72140/diff/1/ Testing --- Tested end-to-end, verified that correct batch size property is now used. Thanks, Andrew Luo
[jira] [Commented] (RANGER-2731) BaseDao.batchCreate uses the wrong property
[ https://issues.apache.org/jira/browse/RANGER-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037959#comment-17037959 ] Andrew Luo commented on RANGER-2731: https://reviews.apache.org/r/72140 > BaseDao.batchCreate uses the wrong property > --- > > Key: RANGER-2731 > URL: https://issues.apache.org/jira/browse/RANGER-2731 > Project: Ranger > Issue Type: Bug > Components: admin > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2731-Use-correct-property-for-batch-size-when-batch-persi.patch > > > BaseDao.batchCreate uses the property ranger.jpa.jdbc.batch-clear.size > instead of the correct property, ranger.jpa.jdbc.batch-persist.size -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2731) BaseDao.batchCreate uses the wrong property
[ https://issues.apache.org/jira/browse/RANGER-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2731: --- Attachment: 0001-RANGER-2731-Use-correct-property-for-batch-size-when-batch-persi.patch > BaseDao.batchCreate uses the wrong property > --- > > Key: RANGER-2731 > URL: https://issues.apache.org/jira/browse/RANGER-2731 > Project: Ranger > Issue Type: Bug > Components: admin > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2731-Use-correct-property-for-batch-size-when-batch-persi.patch > > > BaseDao.batchCreate uses the property ranger.jpa.jdbc.batch-clear.size > instead of the correct property, ranger.jpa.jdbc.batch-persist.size -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2731) BaseDao.batchCreate uses the wrong property
Andrew Luo created RANGER-2731: -- Summary: BaseDao.batchCreate uses the wrong property Key: RANGER-2731 URL: https://issues.apache.org/jira/browse/RANGER-2731 Project: Ranger Issue Type: Bug Components: admin Reporter: Andrew Luo BaseDao.batchCreate uses the property ranger.jpa.jdbc.batch-clear.size instead of the correct property, ranger.jpa.jdbc.batch-persist.size -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72085: RANGER-2721: Fix build instructions in README.txt and Docker build script
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72085/ --- Review request for ranger. Bugs: RANGER-2721 https://issues.apache.org/jira/browse/RANGER-2721 Repository: ranger Description --- README.md still shows the command "mvn clean compile package install assembly:assembly" in the instructions. However, if you run this command, it fails due to a conflict between package and assembly:assembly with the error message "Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default) on project ranger-distro: Failed to create assembly: Error creating assembly archive admin: Problem creating TAR: entry 'ranger-2.1.0-SNAPSHOT-admin/ews/webapp/META-INF/MANIFEST.MF' closed at '0' before the '136' bytes specified in the header were written". This is because after commit https://github.com/apache/ranger/commit/e06abc00667b3935ad86a146be73853fed807eda, the assembly:assembly phase is no longer needed (part of package now) This is a trivial fix to change the README.txt to update the instructions. Diffs - README.txt c1a2f1b96 build_ranger_using_docker.sh 17c8ed0ca Diff: https://reviews.apache.org/r/72085/diff/1/ Testing --- Thanks, Andrew Luo
[jira] [Updated] (RANGER-2721) README.txt shows incorrect build command
[
https://issues.apache.org/jira/browse/RANGER-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Luo updated RANGER-2721:
---
Attachment: 0001-RANGER-2721-Fix-build-instructions-in-README.txt-and.patch
> README.txt shows incorrect build command
>
>
> Key: RANGER-2721
> URL: https://issues.apache.org/jira/browse/RANGER-2721
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: master
> Reporter: Andrew Luo
>Priority: Trivial
> Fix For: master
>
> Attachments:
> 0001-RANGER-2721-Fix-build-instructions-in-README.txt-and.patch
>
>
> README.md still shows the command "{color:#24292e}mvn clean compile package
> install assembly:assembly{color}" in the instructions. However, if you run
> this command, it fails due to a conflict between package and
> assembly:assembly with the error message "Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default) on
> project ranger-distro: Failed to create assembly: Error creating assembly
> archive admin: Problem creating TAR: entry
> 'ranger-2.1.0-SNAPSHOT-admin/ews/webapp/META-INF/MANIFEST.MF' closed at '0'
> before the '136' bytes specified in the header were written". This is
> because after commit
> https://github.com/apache/ranger/commit/e06abc00667b3935ad86a146be73853fed807eda,
> the assembly:assembly phase is no longer needed (part of package now)
> This is a trivial fix to change the README.txt to update the instructions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-2721) README.txt shows incorrect build command
Andrew Luo created RANGER-2721:
--
Summary: README.txt shows incorrect build command
Key: RANGER-2721
URL: https://issues.apache.org/jira/browse/RANGER-2721
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: master
Reporter: Andrew Luo
Fix For: master
README.md still shows the command "{color:#24292e}mvn clean compile package
install assembly:assembly{color}" in the instructions. However, if you run
this command, it fails due to a conflict between package and assembly:assembly
with the error message "Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default) on project
ranger-distro: Failed to create assembly: Error creating assembly archive
admin: Problem creating TAR: entry
'ranger-2.1.0-SNAPSHOT-admin/ews/webapp/META-INF/MANIFEST.MF' closed at '0'
before the '136' bytes specified in the header were written". This is because
after commit
https://github.com/apache/ranger/commit/e06abc00667b3935ad86a146be73853fed807eda,
the assembly:assembly phase is no longer needed (part of package now)
This is a trivial fix to change the README.txt to update the instructions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 72051: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72051/ --- (Updated Jan. 31, 2020, 5:39 a.m.) Review request for ranger. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). Diffs (updated) - security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 85e6aebc1 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6 Diff: https://reviews.apache.org/r/72051/diff/2/ Changes: https://reviews.apache.org/r/72051/diff/1-2/ Testing --- Tested locally on Postgres and MySQL. Tested syntax of SQL on all 5 databases. Thanks, Andrew Luo
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: (was: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch) > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024918#comment-17024918 ] Andrew Luo commented on RANGER-2713: https://reviews.apache.org/r/72051 > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72051: RANGER-2713: Remove audit fields from XXPolicyRef objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72051/ --- Review request for ranger. Bugs: RANGER-2713 https://issues.apache.org/jira/browse/RANGER-2713 Repository: ranger Description --- XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). Diffs - security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 85e6aebc1 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0 security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833 security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6 Diff: https://reviews.apache.org/r/72051/diff/1/ Testing --- Tested locally on Postgres and MySQL. Tested syntax of SQL on all 5 databases. Thanks, Andrew Luo
[jira] [Updated] (RANGER-2713) Remove audit fields from XXPolicyRef objects
[ https://issues.apache.org/jira/browse/RANGER-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2713: --- Attachment: 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > Remove audit fields from XXPolicyRef objects > > > Key: RANGER-2713 > URL: https://issues.apache.org/jira/browse/RANGER-2713 > Project: Ranger > Issue Type: Improvement > Components: admin, audit >Affects Versions: master > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2713-Remove-audit-fields-from-XXPolicyRef-obj.patch > > > XXPolicyRef objects have fields such as create time, update time, added by > user ID, updated by user ID, but there fields are entirely useless since they > are all copied from the XXPolicy object. In addition, while improving > performance for creation of policies with large numbers of users, we > discovered that a lot of time was being spent in JPA converting these Date > objects especially. After removing these fields we saw a significant > performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2713) Remove audit fields from XXPolicyRef objects
Andrew Luo created RANGER-2713: -- Summary: Remove audit fields from XXPolicyRef objects Key: RANGER-2713 URL: https://issues.apache.org/jira/browse/RANGER-2713 Project: Ranger Issue Type: Improvement Components: admin, audit Affects Versions: master Reporter: Andrew Luo XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2643) Use JPA/JDBC batching during policy creation
[ https://issues.apache.org/jira/browse/RANGER-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024912#comment-17024912 ] Andrew Luo commented on RANGER-2643: Committed at https://github.com/apache/ranger/commit/8121961018350014860e29951cbd5b106e8b7923 > Use JPA/JDBC batching during policy creation > > > Key: RANGER-2643 > URL: https://issues.apache.org/jira/browse/RANGER-2643 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 2.0.0 > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2643-Use-JPA-JDBC-batching-during-policy-crea.patch > > > Policy creation is slow if the policy contains a large number of users or > resources. Part of the reason for this is that each row is a single > statement/transaction. > https://reviews.apache.org/r/71746 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 71746: RANGER-2643: Use JPA/JDBC batching during policy creation
> On Nov. 11, 2019, 2:22 a.m., Don Bosco Durai wrote: > > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java > > Lines 46 (patched) > > <https://reviews.apache.org/r/71746/diff/1/?file=2172004#file2172004line46> > > > > Should we take this value from propery? > > > > In RangerBizUtil, we already have a property for batchSize. But it > > might be better to create another one for this. E.g. bulkCreateBatchSize. > > > > Otherwise, code looks good. Thanks for your contribution Fixed, thanks for the review. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71746/#review218592 ----------- On Nov. 16, 2019, 10:46 p.m., Andrew Luo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71746/ > --- > > (Updated Nov. 16, 2019, 10:46 p.m.) > > > Review request for ranger. > > > Bugs: RANGER-2643 > https://issues.apache.org/jira/browse/RANGER-2643 > > > Repository: ranger > > > Description > --- > > Policy creation is slow if the policy contains a large number of users or > resources. Part of the reason for this is that each row is a single > statement/transaction. > > > Diffs > - > > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > 7b2356bfd > security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java > 3761ef2ce > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > 85289dd71 > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java > bdd8fbbb6 > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 54c9ee340 > security-admin/src/main/resources/META-INF/persistence.xml 211715978 > security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml > 99162971c > > > Diff: https://reviews.apache.org/r/71746/diff/2/ > > > Testing > --- > > Compilation/end-to-end test of policy creation. > > > Thanks, > > Andrew Luo > >
[jira] [Updated] (RANGER-2643) Use JPA/JDBC batching during policy creation
[ https://issues.apache.org/jira/browse/RANGER-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2643: --- Attachment: (was: 0001-RANGER-2643-Use-JPA-JDBC-batching-during-policy-crea.patch) > Use JPA/JDBC batching during policy creation > > > Key: RANGER-2643 > URL: https://issues.apache.org/jira/browse/RANGER-2643 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 2.0.0 > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2643-Use-JPA-JDBC-batching-during-policy-crea.patch > > > Policy creation is slow if the policy contains a large number of users or > resources. Part of the reason for this is that each row is a single > statement/transaction. > https://reviews.apache.org/r/71746 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2643) Use JPA/JDBC batching during policy creation
[ https://issues.apache.org/jira/browse/RANGER-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Luo updated RANGER-2643: --- Attachment: 0001-RANGER-2643-Use-JPA-JDBC-batching-during-policy-crea.patch > Use JPA/JDBC batching during policy creation > > > Key: RANGER-2643 > URL: https://issues.apache.org/jira/browse/RANGER-2643 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 2.0.0 > Reporter: Andrew Luo >Priority: Minor > Attachments: > 0001-RANGER-2643-Use-JPA-JDBC-batching-during-policy-crea.patch > > > Policy creation is slow if the policy contains a large number of users or > resources. Part of the reason for this is that each row is a single > statement/transaction. > https://reviews.apache.org/r/71746 -- This message was sent by Atlassian Jira (v8.3.4#803005)
