----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39161/#review103140 -----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java (line 128) <https://reviews.apache.org/r/39161/#comment161086> Seems like there is some duplication of code. Can we change the run() method like this If you feel it avoids code duplication? public run() { try { //purge result set paths purgePaths(resultsetPath, true, resultsetRetention); //purge hdfsoutputpaths purgePaths(hdfsOutputPath,hdfsOutputRetention); } private void purgePaths(Path outputPath, boolean isFile, DateUtil.TimeDiff retention) { int counter = 0; FileSystem fs = outputPath.getFileSystem(conf); FileStatus[] fileList = fs.listStatus(resultsetPath); for (FileStatus f : fileList) { if (f.isFile() && canBePurged(f, retention)) { try { fs.delete(f.getPath()); counter++; } catch (IOException e) { getMetrics().incrCounter(this.getClass(), QUERY_RESULT_PURGER_ERROR_COUNTER); } } } log.info("Purged {} files in {}", counter, resultsetPath.toString()); } - Raju Bairishetti On Oct. 19, 2015, 12:22 p.m., Deepak Barr wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39161/ > ----------------------------------------------------------- > > (Updated Oct. 19, 2015, 12:22 p.m.) > > > Review request for lens. > > > Bugs: LENS-292 > https://issues.apache.org/jira/browse/LENS-292 > > > Repository: lens > > > Description > ------- > > Added query result purge policy on both Driver persisted and Lens persisted > resultset. Default retention of 1 day. Purging can be turned off with a > setting. > > > Diffs > ----- > > lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > f202603 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > deb8572 > > lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java > PRE-CREATION > lens-server/src/main/resources/lensserver-default.xml 1e6bd10 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java > PRE-CREATION > src/site/apt/admin/config.apt 3a0678f > > Diff: https://reviews.apache.org/r/39161/diff/ > > > Testing > ------- > > Yes. > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.785 > s] > [INFO] Lens ............................................... SUCCESS [ 5.339 > s] > [INFO] Lens API ........................................... SUCCESS [ 28.950 > s] > [INFO] Lens API for server and extensions ................. SUCCESS [ 28.187 > s] > [INFO] Lens Cube .......................................... SUCCESS [07:05 > min] > [INFO] Lens DB storage .................................... SUCCESS [ 28.169 > s] > [INFO] Lens Query Library ................................. SUCCESS [ 22.031 > s] > [INFO] Lens Hive Driver ................................... SUCCESS [04:13 > min] > [INFO] Lens Driver for JDBC ............................... SUCCESS [ 49.452 > s] > [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 22.865 > s] > [INFO] Lens Server ........................................ SUCCESS [10:58 > min] > [INFO] Lens client ........................................ SUCCESS [ 47.198 > s] > [INFO] Lens CLI ........................................... SUCCESS [03:44 > min] > [INFO] Lens Examples ...................................... SUCCESS [ 11.516 > s] > [INFO] Lens Distribution .................................. SUCCESS [ 12.779 > s] > [INFO] Lens ML Lib ........................................ SUCCESS [02:00 > min] > [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.212 > s] > [INFO] Lens Regression .................................... SUCCESS [ 15.624 > s] > [INFO] Lens UI ............................................ SUCCESS [ 3.415 > s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 32:45 min > [INFO] Finished at: 2015-10-07T23:19:44+05:30 > [INFO] Final Memory: 218M/2795M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Deepak Barr > >
