----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74568/#review225702 -----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java Lines 4104 (patched) <https://reviews.apache.org/r/74568/#comment314147> Consider maintaining a map of serviceName => serviceType, which can help avoid repeated database calls here. Map<String, String> svcNameToSvcType = new HashMap<>(); for (RangerPolicy policy : policies) { String serviceType = policy.getServiceType(); if (StringUtils.isBlank(serviceType)) { serviceType = svcNameToSvcType.get(policy.getService); if (StringUtils.isBlank(serviceType)) { XXService xxservice = daoMgr.getXXService().findByName(policy.getService()); XXServiceDef xxservDef = xxservice != null ? daoMgr.getXXServiceDef().getById(xxservice.getType()) : null; serviceType = xxservDef != null ? xxservDef.getName() : null; if (StringUtils.isNotBlank(serviceType)) { svcNameToSvcType.put(policy.getService(), serviceType); } } } ... } security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java Line 4121 (original), 4116 (patched) <https://reviews.apache.org/r/74568/#comment314148> Consider sending svcNameToSvcType to writeBookForPolicyItems() calls, and update this method to replace database calls with lookup in svcNameToSvcType. - Madhan Neethiraj On Aug. 28, 2023, 9 a.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74568/ > ----------------------------------------------------------- > > (Updated Aug. 28, 2023, 9 a.m.) > > > Review request for ranger, Abhishek Kumar, bhavik patel, Dhaval Shah, > Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, > Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-4382 > https://issues.apache.org/jira/browse/RANGER-4382 > > > Repository: ranger > > > Description > ------- > > **Problem Statement:** Ranger export CSV rest API taking more than 4 hours to > complete the request for just 60k policies > > **Proposed solution:** Ranger exportCSV REST endpoint returns RangerPolicies > along with Policy's service type, to populate service types additional DB > call to x_service and x_service_def table is made, though its possible that > initially fetched RangerPolicy objects may already have serviceType which can > be utilized if its available rather making the db call directly. > > > Diffs > ----- > > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > 8f1174ac4 > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 86959423c > > > Diff: https://reviews.apache.org/r/74568/diff/1/ > > > Testing > ------- > > Curl request without patch with 64k policies took 4 hours: > > curl -ivk -u admin:Admin123 -o /tmp/exportcsv.csv -X GET > 'http://localhost:6182/service/plugins/policies/csv' > > Curl request with the proposed patch with 64k policies took 15 seconds in the > first attempt and 10 seconds in the second attempt. > > curl -ivk -u admin:Admin123 -o /tmp/exportcsv.csv -X GET > 'http://localhost:6182/service/plugins/policies/csv' > > > Thanks, > > Pradeep Agrawal > >
