----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37346/#review99955 -----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 789) <https://reviews.apache.org/r/37346/#comment157006> Taking care of LENS-783 here. - Rajat Khandelwal On Sept. 22, 2015, 2:55 p.m., Rajat Khandelwal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37346/ > ----------------------------------------------------------- > > (Updated Sept. 22, 2015, 2:55 p.m.) > > > Review request for lens. > > > Bugs: LENS-693 > https://issues.apache.org/jira/browse/LENS-693 > > > Repository: lens > > > Description > ------- > > Changes include: > > 1. The configuration `lens.server.max.finished.queries` has been taken off. > Now All queries are purged as soon as they are purgable. The logic of > purgability of a query is that: > 1. if the query failed, it's purgable > 2. if the query doesn't have a result, it's purgable > 3. If it has a result, then result class is asked whether it's purgable > or not. This is the second layer of the logic. Current logic in that layer is > as follows: > 1. If the result is persistent, it's purgable > 2. If the result is in memory, it's not purgable initially, but > after all the rows have been accessed by someone, it becomes purgable. The > rows can be accessed in the following ways: > 1. While sending Email notification, in memory result is > appended in the email. Hence rows are accessed. > 2. Explicit call by a user for reading result set (/resultset > REST API, but not the /httpresultset API, http results are not available for > in memory results.) > > > 2. There is a new configuration `lens.server.purge.interval`. The value is > supposed to be in milliseconds. This is the frequency of purger thread. > 3. Purger thread had a bug in which it had a queue of finished queries which > were inserted according to their finish time. Purging of later queries was > dependent on purging of earlier queries. If a later query becomes purgable, > it's not purged untill all the queries before it become purgable and are > purged. To fix this, purger thread now runs periodically and scans the whole > queue, purges whichever ones are purgable. > 4. FinishedLensQuery is a model object, but has a field which is not a field > in corresponding database. That field is removed from equals check > 5. `instanceof` Checks have been converted to polymorphism: > 1. For http result set availability. > 2. For forming email message of the result of a finished query. > 6. Email notifications are now sent irrespective of whether the query has > been purged or not. Earlier there was a bug that if a query is purged, email > notification for that isn't sent. > 7. Taking care of LENS-783 > 8. Modularizing testing of email notifications > 9. Made more util methods in RestAPITestUtil which became re-usable in server > tests like TestQueryService. > 10. Added Test cases for query purging. > > > Diffs > ----- > > lens-api/src/main/java/org/apache/lens/api/LensConf.java > 3b1ad87ffe16b572437b92d90c793ea1881c2e0d > lens-api/src/main/java/org/apache/lens/api/query/InMemoryQueryResult.java > 6a9f4558afc3a8430b43dcc65d73c813575a334d > lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java > af439ff58002b06e2631585d35dfe9754ce84d4e > lens-api/src/main/java/org/apache/lens/api/query/PersistentQueryResult.java > a2e10c6560aaa551a15408b2937c3d495b51af5f > lens-api/src/main/java/org/apache/lens/api/query/QueryResult.java > 3ecf0c229494dd82a531bc5a310e3b056a86d451 > lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java > 3c8531feb40276e076622b32f15c2f3d7ee782b2 > lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java > 006eaedcd5cdf86b9aae6c4bbfcd92ba6c1df8ba > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 31c343a7b72c13ff9eb9b01038574297629e381b > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HivePersistentResultSet.java > 00e1e53444ca76fa5096efc69fc4bdfdd16d116a > > lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java > 78b3320977f9b46b2719ba80190fa77e87ecc729 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > 096d26ee669c7e8c30df134ac334877311c1812e > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java > 3b761268645acf150d149e51f69521bf164c61ee > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensResultSet.java > 929a3026da9d9789b64ab1f04fdd063db3443432 > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/PersistentResultSet.java > bbde1705fec7fb946708b17d57eb5ad74fcb7cef > > lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java > d8c04dbdfdbf5309ef4ec6b07de27de33d2ff517 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java > 12de0a51b33a5ca09cd494f2ea842794b800f9e6 > > lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java > b28669e9070578e5121eb4b1c59e8e3ac7cb83de > > lens-server/src/main/java/org/apache/lens/server/query/LensPersistentResult.java > b65a5f4ac661f060b51d5ee1c899dfd93aa8be90 > > lens-server/src/main/java/org/apache/lens/server/query/QueryEndNotifier.java > 45ba7ac0300cc7b1d5d2ab723c4adf8d60801e11 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > 9e27dd427cd573f57b06a3c8157c49f723b54caf > lens-server/src/main/java/org/apache/lens/server/query/ResultFormatter.java > af42eb06e2c030a7040b970e982cab782b2bb251 > lens-server/src/main/resources/lensserver-default.xml > 3c8fc09242bb3f873aa43630c9f386435f10c659 > lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java > 05bd99f8819477c9560e231148b1e9ef7a8f061b > > lens-server/src/test/java/org/apache/lens/server/common/RestAPITestUtil.java > 0945598eed81515702402e9933b2592186209508 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryEndEmailNotifier.java > 82ce205c8d54c6665262ee3baf8c521203743eb5 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java > fa8d6eed0b98e9debe5e1c6f138bb7b104dff135 > > lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java > 98a658e9c02a9da9d6708e4db1df161c6ea1a784 > lens-server/src/test/resources/lens-site.xml > f728810867b677b6c7fba29229206eed1937e37f > > Diff: https://reviews.apache.org/r/37346/diff/ > > > Testing > ------- > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules ............................. SUCCESS [1.828s] > [INFO] Lens .............................................. SUCCESS [2.893s] > [INFO] Lens API .......................................... SUCCESS [22.171s] > [INFO] Lens API for server and extensions ................ SUCCESS [21.408s] > [INFO] Lens Cube ......................................... SUCCESS [5:06.448s] > [INFO] Lens DB storage ................................... SUCCESS [22.723s] > [INFO] Lens Query Library ................................ SUCCESS [17.844s] > [INFO] Lens Hive Driver .................................. SUCCESS [3:23.430s] > [INFO] Lens Driver for JDBC .............................. SUCCESS [41.430s] > [INFO] Lens Elastic Search Driver ........................ SUCCESS [22.646s] > [INFO] Lens Server ....................................... SUCCESS [8:01.584s] > [INFO] Lens client ....................................... SUCCESS [37.284s] > [INFO] Lens CLI .......................................... SUCCESS [2:35.185s] > [INFO] Lens Examples ..................................... SUCCESS [9.901s] > [INFO] Lens Distribution ................................. SUCCESS [8.453s] > [INFO] Lens ML Lib ....................................... SUCCESS [1:19.537s] > [INFO] Lens ML Ext Distribution .......................... SUCCESS [2.206s] > [INFO] Lens Regression ................................... SUCCESS [10.486s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 24:08.364s > [INFO] Finished at: Tue Sep 22 08:33:26 UTC 2015 > [INFO] Final Memory: 184M/1305M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Rajat Khandelwal > >