----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74552/#review225670 -----------------------------------------------------------
Ship it! Ship It! - Ramachandran Krishnan On Aug. 21, 2023, 1:46 p.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74552/ > ----------------------------------------------------------- > > (Updated Aug. 21, 2023, 1:46 p.m.) > > > Review request for ranger, bhavik patel, Abhay Kulkarni, Madhan Neethiraj, > Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja > Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-4353 > https://issues.apache.org/jira/browse/RANGER-4353 > > > Repository: ranger > > > Description > ------- > > **Problem Statement:** Currently ranger transaction entries are being stored > in x_trx_log table which may have lot of entries in few days. User need to > manually remove the entries from x_trx_log table time to time in order to > maintain disk space or handle disk space issues in a production env. > > > ** Proposed Solution: ** > > > Option-1: Delete the entries during every start of ranger-admin service: > > > This patch exposes two ranger configs > 1) "ranger.admin.init.purge.transaction_records" => should be set to 'true'. > default is false. > 2) "ranger.admin.init.purge.transaction_records.retention.days" => which > accepts positive numerical values in days. > > > According to above configs During the start of ranger-admin x_trx_log table > entries older than the mentioned days shall be removed. > > > When "ranger.admin.init.purge.transaction_records" is set to 'true' and > "ranger.admin.init.purge.transaction_records.retention.days" value set to a > positive number this feature shall be affective. > > > Option-2: : User can call below mentioned REST api to delete the records. > User must use a credential which has admin role in the ranger to call this > REST API. > > > curl -u admin:admin -H "Accept: application/json" -H "Content-Type: > application/json" -X DELETE > 'http://localhost:6080/service/public/v2/api/server/purge/records?type=trx_records&retentionDays=5' > > > if retentionDays parameter is not provided then default value 180 shall be > considered. > > > Note: The proposed implementation shall not delete entries every day as there > is no daemon process shall be running at the background, hence deletion of > entries shall be attempted only during the start of ranger. > > > Diffs > ----- > > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > ed1ea0376 > security-admin/src/main/java/org/apache/ranger/db/XXAuthSessionDao.java > f69b8d2bb > security-admin/src/main/java/org/apache/ranger/db/XXTrxLogDao.java > a83e91f5b > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > d2d76733e > security-admin/src/main/resources/META-INF/jpa_named_queries.xml 1e8e4e2c5 > security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d6bf174e9 > > > Diff: https://reviews.apache.org/r/74552/diff/2/ > > > Testing > ------- > > **Approach-1:** ranger-admin start approach > > Added "ranger.admin.init.purge.login_records" and > "ranger.admin.init.purge.login_records.retention.days" in > ranger-admin-site.xml with value 30. > Added "ranger.admin.init.purge.transaction_records" and > "ranger.admin.init.purge.transaction_records.retention.days" in > ranger-admin-site.xml with value 31. > Restarted the ranger-admin. > > Logs: > > 2023-08-22 09:56:19,595 INFO org.apache.ranger.db.XXAuthSessionDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleting x_auth_sess records > that are older than 30 days, that is, older than Sun Jul 23 09:56:19 UTC 2023 > 2023-08-22 09:56:19,893 INFO org.apache.ranger.db.XXAuthSessionDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleted 3743 x_auth_sess records > 2023-08-22 09:56:19,893 INFO org.apache.ranger.db.XXAuthSessionDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Updating x_trx_log.sess_id with > null which are older than 30 days, that is, older than Sun Jul 23 09:56:19 > UTC 2023 > 2023-08-22 09:56:19,903 INFO org.apache.ranger.db.XXAuthSessionDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Updated 9 x_trx_log records > 2023-08-22 09:56:19,903 INFO org.apache.ranger.biz.ServiceDBStore: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleted 3743 records from > x_auth_sess that are older than 30 days > 2023-08-22 09:56:19,920 INFO org.apache.ranger.db.XXTrxLogDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleting x_trx_log records that > are older than 31 days, that is, older than Sat Jul 22 09:56:19 UTC 2023 > 2023-08-22 09:56:19,924 INFO org.apache.ranger.db.XXTrxLogDao: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleted 9 x_trx_log records > 2023-08-22 09:56:19,924 INFO org.apache.ranger.biz.ServiceDBStore: > [myhost-1.myhost.root.hwx.site-startStop-1]: Deleted 9 records from x_trx_log > that are older than 31 days > > > **Approach-2: REST API approach** > > > curl -u admin:admin -H "Accept: application/json" -H "Content-Type: > application/json" -X DELETE > 'http://localhost:6080/service/public/v2/api/server/purge/records?type=trx_records&retentionDays=20' > > > request completed successfully with response code 204. Checked the entries in > x_trx_log table manually and x_trx_log entries older than 20 days were > deleted. > below log was printed in the ranger log file: > > > 2023-08-22 10:05:39,735 INFO org.apache.ranger.db.XXTrxLogDao: > [http-nio-6080-exec-51]: Deleting x_trx_log records that are older than 20 > days, that is, older than Wed Aug 02 10:05:39 UTC 2023 > 2023-08-22 10:05:39,740 INFO org.apache.ranger.db.XXTrxLogDao: > [http-nio-6080-exec-51]: Deleted 4 x_trx_log records > 2023-08-22 10:05:39,740 INFO org.apache.ranger.biz.ServiceDBStore: > [http-nio-6080-exec-51]: Deleted 4 records from x_trx_log that are older than > 20 days > > > Note: Below is the SQL statement to check the x_trx_log table records: > select date(create_time),count(*) from x_trx_log group by date(create_time); > > > Thanks, > > Pradeep Agrawal > >
