Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-20 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 20, 2015, 8:05 a.m.)


Review request for lens.


Changes
---

1. Increment error counter when delete() call returns false.
2. Reused path deletion code.


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 deb8572 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
  
lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 PRE-CREATION 
  src/site/apt/admin/config.apt 3a0678f 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Deepak Barr 


> On Oct. 17, 2015, 3:24 a.m., Raju Bairishetti wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java,
> >  line 983
> > 
> >
> > @Deepak, Is there any specific reason behind for not accepting the 
> > value in ***secs**?

The only reason being - "1 day" is more readable than "86400" seconds. For 
retention config, Some users may want to use bigger time units and some may use 
small time units. This kind of input provides makes it easy. Lets say if a user 
wants to have a retention for 2 months...then, he can simple put 2 months 
instead of 5184000 seconds


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103010
---


On Oct. 16, 2015, 7:43 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 7:43 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Deepak Barr 


> On Oct. 17, 2015, 3:09 a.m., Raju Bairishetti wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java,
> >  line 129
> > 
> >
> > Log won't be printed in case of exceptions while deleting paths. Assume 
> > like deleted 10 directories out of 20. Then we do not have for deleted 10 
> > directories from hdfsoutput path.
> > 
> > I feel metric counter for the purged files/dirs would be helpful. Any 
> > thoughts?

I agree. I have added a metric counter as per amareshwari's suggestion. Will 
use it for path deletion exceptions too.


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103009
---


On Oct. 16, 2015, 7:43 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 7:43 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 19, 2015, 11:33 a.m.)


Review request for lens.


Changes
---

1. Added metrics counter to handle purge exceptions
2. Removed the code to close FileSystem object.
3. Other small suggested changes.


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 deb8572 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
  
lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 PRE-CREATION 
  src/site/apt/admin/config.apt 3a0678f 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 19, 2015, 12:22 p.m.)


Review request for lens.


Changes
---

Fixed purger counter variable.


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 deb8572 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
  
lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 PRE-CREATION 
  src/site/apt/admin/config.apt 3a0678f 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103121
---



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 74)


Can we rename variable to QUERY_RESULT_PURGER_ERROR_COUNTER


- Amareshwari Sriramadasu


On Oct. 19, 2015, 11:33 a.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 19, 2015, 11:33 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  deb8572 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Puneet Gupta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103129
---



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 119)


recursive delete can be enabled.



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 122)


apart from IOException , fs.delete() returns a boolean we should be 
accounted for while undating counters (error/success).


- Puneet Gupta


On Oct. 19, 2015, 12:22 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 19, 2015, 12:22 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  deb8572 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-19 Thread Deepak Barr 


> On Oct. 19, 2015, 2:53 p.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java,
> >  line 119
> > 
> >
> > recursive delete can be enabled.

I guess in case of deletion of file, this flag can be either of true/false. 
https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#delete(org.apache.hadoop.fs.Path,
 boolean).


- Deepak


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103129
---


On Oct. 19, 2015, 12:22 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 19, 2015, 12:22 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  deb8572 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-16 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 16, 2015, 7:43 p.m.)


Review request for lens.


Changes
---

1. Made the retention time unit flexible.
2. Added healthcheck for resultpurger thread.


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 d57812f 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
  
lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 PRE-CREATION 
  src/site/apt/admin/config.apt 3a0678f 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-16 Thread Puneet Gupta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102899
---

Ship it!


Ship It!

- Puneet Gupta


On Oct. 16, 2015, 9:44 a.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 9:44 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-16 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102903
---

Ship it!



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (lines 1139 - 1140)


Constructor can take conf instead of a separate method.


- Rajat Khandelwal


On Oct. 16, 2015, 3:14 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 3:14 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-16 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102900
---



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(lines 968 - 973)


Instead of keeping the unit of retention time fixed(currently, DAYS), let's 
leave this to the user. You can use this: 
`org.apache.lens.cube.parse.DateUtil.TimeDiff#parseFrom`



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 1098)


add purger thread to health check of execution service.


- Rajat Khandelwal


On Oct. 16, 2015, 3:14 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 3:14 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-16 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review103002
---



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 131)


Can we increment an error counter for failures? Should we catch all 
Exception for thread not to die?



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 135)


We should not call fs.close() here. fs.close() will make all other clients 
to file system also get closed. So, any application should not call fs.close 
otherthan shutting down.



lens-server/src/main/resources/lensserver-default.xml (line 283)


Default value here and the code seems different



lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 (line 80)


20 seconds is too much time for the test to finish, can  we change purge 
interval to 1 second for test and wait for 2 seconds?


- Amareshwari Sriramadasu


On Oct. 16, 2015, 7:43 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 16, 2015, 7:43 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DateUtil.java cbcc76c 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-15 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 15, 2015, 7:53 p.m.)


Review request for lens.


Changes
---

1. Refactored query result purger initialization.
2. Add new retention property for hdfs output path
3. Added test case for resultset purger.
4. Updated config.apt
5. Few other changes as per suggestions


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 73179e1 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
  
lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
 PRE-CREATION 
  src/site/apt/admin/config.apt 3a0678f 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-15 Thread Puneet Gupta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102869
---



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 58)


Query: Do we need to provide comments for all fields ? I am not sure what 
is the general norm



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 76)


Is it possible to add name for this thread? It will be useful while 
debugging issues and taking thread dumps .



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 79)


Can we add a info log? it can also specify purge interval and retention 
periods



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 98)


Should we log all deleted file names for book-keeking purpose ? If that 
seems like an overkill, we can keep a counter for deleted files and log the 
counter value also for both local and hdfs. 

If counter is zero, we can log, found no files to purge

Logging file names may be better from auditing point of view.


- Puneet Gupta


On Oct. 15, 2015, 8:03 p.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 15, 2015, 8:03 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  d57812f 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
>   
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryResultPurger.java
>  PRE-CREATION 
>   src/site/apt/admin/config.apt 3a0678f 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-14 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102732
---



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 958)


Can you name config to reflect the time unit as well ?



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 1134)


Shutdown the executor in stop ?



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 40)


Please add a unit test for the purger.



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 49)


make it final



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(lines 62 - 63)


Can these moved to init?



lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
(line 75)


Can we have a different retention for hdfs output path? Once the result is 
formatted and pesristed by server, hdfs outputs can be purged earlier than 
final results.



lens-server/src/main/resources/lensserver-default.xml (line 288)


Can you update apt files as well?


- Amareshwari Sriramadasu


On Oct. 13, 2015, 11:48 a.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 13, 2015, 11:48 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  73179e1 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
> PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-13 Thread Deepak Barr 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/
---

(Updated Oct. 13, 2015, 11:48 a.m.)


Review request for lens.


Changes
---

Updated the diff with the suggestions.

1. Took the ResultPurger class outside of QueryExecutionServiceImpl
2. By default, purge is disabled.
3. Fixed comments.


Bugs: LENS-292
https://issues.apache.org/jira/browse/LENS-292


Repository: lens


Description
---

Added query result purge policy on both Driver persisted and Lens persisted 
resultset. Default retention of 1 day. Purging can be turned off with a setting.


Diffs (updated)
-

  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
f202603 
  
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 73179e1 
  lens-server/src/main/java/org/apache/lens/server/query/QueryResultPurger.java 
PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 1e6bd10 

Diff: https://reviews.apache.org/r/39161/diff/


Testing
---

Yes.

[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 s]
[INFO] Lens ... SUCCESS [  5.339 s]
[INFO] Lens API ... SUCCESS [ 28.950 s]
[INFO] Lens API for server and extensions . SUCCESS [ 28.187 s]
[INFO] Lens Cube .. SUCCESS [07:05 min]
[INFO] Lens DB storage  SUCCESS [ 28.169 s]
[INFO] Lens Query Library . SUCCESS [ 22.031 s]
[INFO] Lens Hive Driver ... SUCCESS [04:13 min]
[INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 s]
[INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 s]
[INFO] Lens Server  SUCCESS [10:58 min]
[INFO] Lens client  SUCCESS [ 47.198 s]
[INFO] Lens CLI ... SUCCESS [03:44 min]
[INFO] Lens Examples .. SUCCESS [ 11.516 s]
[INFO] Lens Distribution .. SUCCESS [ 12.779 s]
[INFO] Lens ML Lib  SUCCESS [02:00 min]
[INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 s]
[INFO] Lens Regression  SUCCESS [ 15.624 s]
[INFO] Lens UI  SUCCESS [  3.415 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 32:45 min
[INFO] Finished at: 2015-10-07T23:19:44+05:30
[INFO] Final Memory: 218M/2795M
[INFO] 


Thanks,

Deepak Barr

Re: Review Request 39161: LENS-292 : Query result retention policy

2015-10-09 Thread Puneet Gupta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39161/#review102023
---



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 951)


comment seems wrong. please check



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 953)


May be result Set Purging should be false by default. Some people might be 
caught offguard otherwise.



lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
(line 971)


comment seems wrong. please check



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 130)


Can chnage to 7*millisInDay now . Just a suggestion .



lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
 (line 1018)


Wondering that QueryExecutionServiceImpl is already close to 3k lines. 
Should we create a separate class for QueryResultPurger. 

QueryExecutionServiceImpl can just call QueryResultPurger.init(conf). 
QueryResultPurger will have all the logic and memebers/varaibles specific to 
result set purging 

We might have to move other private class also out later (if we get some 
time to refactor)


- Puneet Gupta


On Oct. 9, 2015, 7:31 a.m., Deepak Barr wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39161/
> ---
> 
> (Updated Oct. 9, 2015, 7:31 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-292
> https://issues.apache.org/jira/browse/LENS-292
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Added query result purge policy on both Driver persisted and Lens persisted 
> resultset. Default retention of 1 day. Purging can be turned off with a 
> setting.
> 
> 
> Diffs
> -
> 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  f202603 
>   
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
>  73179e1 
>   lens-server/src/main/resources/lensserver-default.xml 1e6bd10 
> 
> Diff: https://reviews.apache.org/r/39161/diff/
> 
> 
> Testing
> ---
> 
> Yes.
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .. SUCCESS [  2.785 
> s]
> [INFO] Lens ... SUCCESS [  5.339 
> s]
> [INFO] Lens API ... SUCCESS [ 28.950 
> s]
> [INFO] Lens API for server and extensions . SUCCESS [ 28.187 
> s]
> [INFO] Lens Cube .. SUCCESS [07:05 
> min]
> [INFO] Lens DB storage  SUCCESS [ 28.169 
> s]
> [INFO] Lens Query Library . SUCCESS [ 22.031 
> s]
> [INFO] Lens Hive Driver ... SUCCESS [04:13 
> min]
> [INFO] Lens Driver for JDBC ... SUCCESS [ 49.452 
> s]
> [INFO] Lens Elastic Search Driver . SUCCESS [ 22.865 
> s]
> [INFO] Lens Server  SUCCESS [10:58 
> min]
> [INFO] Lens client  SUCCESS [ 47.198 
> s]
> [INFO] Lens CLI ... SUCCESS [03:44 
> min]
> [INFO] Lens Examples .. SUCCESS [ 11.516 
> s]
> [INFO] Lens Distribution .. SUCCESS [ 12.779 
> s]
> [INFO] Lens ML Lib  SUCCESS [02:00 
> min]
> [INFO] Lens ML Ext Distribution ... SUCCESS [  3.212 
> s]
> [INFO] Lens Regression  SUCCESS [ 15.624 
> s]
> [INFO] Lens UI  SUCCESS [  3.415 
> s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 32:45 min
> [INFO] Finished at: 2015-10-07T23:19:44+05:30
> [INFO] Final Memory: 218M/2795M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Deepak Barr
> 
>