> On Sept. 22, 2015, 10:23 a.m., Amareshwari Sriramadasu wrote: > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java, > > line 779 > > <https://reviews.apache.org/r/37346/diff/9/?file=1080331#file1080331line779> > > > > Let us change the name to lens.server.query.purger.interval
If the name is lens.server.query.purge.interval, it seems it is still linked to single query. Instead shall we rename it to lens.server.querypurger.sleep.interval ? Please update apt file as well. - Amareshwari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37346/#review99966 ----------------------------------------------------------- On Sept. 22, 2015, 2:46 p.m., Rajat Khandelwal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37346/ > ----------------------------------------------------------- > > (Updated Sept. 22, 2015, 2:46 p.m.) > > > Review request for lens. > > > Bugs: LENS-693 > https://issues.apache.org/jira/browse/LENS-693 > > > Repository: lens > > > Description > ------- > > Changes include: > > 1. The configuration `lens.server.max.finished.queries` has been taken off. > Now All queries are purged as soon as they are purgable. The logic of > purgability of a query is that: > 1. if the query failed, it's purgable > 2. if the query doesn't have a result, it's purgable > 3. If it has a result, then result class is asked whether it's purgable > or not. This is the second layer of the logic. Current logic in that layer is > as follows: > 1. If the result is persistent, it's purgable > 2. If the result is in memory, it's not purgable initially, but > after all the rows have been accessed by someone, it becomes purgable. The > rows can be accessed in the following ways: > 1. While sending Email notification, in memory result is > appended in the email. Hence rows are accessed. > 2. Explicit call by a user for reading result set (/resultset > REST API, but not the /httpresultset API, http results are not available for > in memory results.) > > > 2. There is a new configuration `lens.server.purge.interval`. The value is > supposed to be in milliseconds. This is the frequency of purger thread. > 3. Purger thread had a bug in which it had a queue of finished queries which > were inserted according to their finish time. Purging of later queries was > dependent on purging of earlier queries. If a later query becomes purgable, > it's not purged untill all the queries before it become purgable and are > purged. To fix this, purger thread now runs periodically and scans the whole > queue, purges whichever ones are purgable. > 4. FinishedLensQuery is a model object, but has a field which is not a field > in corresponding database. That field is removed from equals check > 5. `instanceof` Checks have been converted to polymorphism: > 1. For http result set availability. > 2. For forming email message of the result of a finished query. > 6. Email notifications are now sent irrespective of whether the query has > been purged or not. Earlier there was a bug that if a query is purged, email > notification for that isn't sent. > 7. Taking care of LENS-783 > 8. Modularizing testing of email notifications > 9. Made more util methods in RestAPITestUtil which became re-usable in server > tests like TestQueryService. > 10. Added Test cases for query purging. > 11. The way we're taking care of purge happening before event handlers are > done with context is by utilizing garbage collection. We pass context to the > QueryEnded event object. Now whenever the event happens, it has access to the > context. It has access even when purger has purged the query(since the object > will only go obsolete if no one has a handle to it). In purging, the context > is removed from various query collections, but context object is intact. > Whenever all event handlers for the QueryEnded event are done, both the event > object and the query context object will be garbage collected. > > > Diffs > ----- > > lens-api/src/main/java/org/apache/lens/api/LensConf.java > 3b1ad87ffe16b572437b92d90c793ea1881c2e0d > lens-api/src/main/java/org/apache/lens/api/query/InMemoryQueryResult.java > 6a9f4558afc3a8430b43dcc65d73c813575a334d > lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java > af439ff58002b06e2631585d35dfe9754ce84d4e > lens-api/src/main/java/org/apache/lens/api/query/PersistentQueryResult.java > a2e10c6560aaa551a15408b2937c3d495b51af5f > lens-api/src/main/java/org/apache/lens/api/query/QueryResult.java > 3ecf0c229494dd82a531bc5a310e3b056a86d451 > lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java > 3c8531feb40276e076622b32f15c2f3d7ee782b2 > lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java > 006eaedcd5cdf86b9aae6c4bbfcd92ba6c1df8ba > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 31c343a7b72c13ff9eb9b01038574297629e381b > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HivePersistentResultSet.java > 00e1e53444ca76fa5096efc69fc4bdfdd16d116a > > lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java > 78b3320977f9b46b2719ba80190fa77e87ecc729 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > 096d26ee669c7e8c30df134ac334877311c1812e > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java > 3b761268645acf150d149e51f69521bf164c61ee > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensResultSet.java > 929a3026da9d9789b64ab1f04fdd063db3443432 > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/PersistentResultSet.java > bbde1705fec7fb946708b17d57eb5ad74fcb7cef > > lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java > d8c04dbdfdbf5309ef4ec6b07de27de33d2ff517 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryCancelled.java > bc0465c782a86d484747f0375135191bef0c49d1 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryClosed.java > ea8f70fc54b7c646e982a4de882eea92124b73d5 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java > 12de0a51b33a5ca09cd494f2ea842794b800f9e6 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryEnded.java > 3e9474c642827e401a1e243ae25d328557f1adc0 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryFailed.java > 40a1c0f1078c82f882eab60a0f0b20b501789da5 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QuerySuccess.java > c551dbab463dc7dd199f4fa1c6c56c956f56a704 > > lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java > b28669e9070578e5121eb4b1c59e8e3ac7cb83de > > lens-server/src/main/java/org/apache/lens/server/query/LensPersistentResult.java > b65a5f4ac661f060b51d5ee1c899dfd93aa8be90 > > lens-server/src/main/java/org/apache/lens/server/query/QueryEndNotifier.java > 45ba7ac0300cc7b1d5d2ab723c4adf8d60801e11 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > 9e27dd427cd573f57b06a3c8157c49f723b54caf > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionStatisticsGenerator.java > b57bc640bee018c2b7da560c2c4bcad5e228d41a > lens-server/src/main/java/org/apache/lens/server/query/ResultFormatter.java > af42eb06e2c030a7040b970e982cab782b2bb251 > lens-server/src/main/resources/lensserver-default.xml > 3c8fc09242bb3f873aa43630c9f386435f10c659 > lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java > 05bd99f8819477c9560e231148b1e9ef7a8f061b > > lens-server/src/test/java/org/apache/lens/server/common/RestAPITestUtil.java > 0945598eed81515702402e9933b2592186209508 > > lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java > b8012be1c30b544e51228c4b9b8a2a7d8281d3bc > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryEndEmailNotifier.java > 82ce205c8d54c6665262ee3baf8c521203743eb5 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java > fa8d6eed0b98e9debe5e1c6f138bb7b104dff135 > > lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java > 98a658e9c02a9da9d6708e4db1df161c6ea1a784 > lens-server/src/test/resources/lens-site.xml > f728810867b677b6c7fba29229206eed1937e37f > > Diff: https://reviews.apache.org/r/37346/diff/ > > > Testing > ------- > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Lens Checkstyle Rules ............................. SUCCESS [1.844s] > [INFO] Lens .............................................. SUCCESS [3.076s] > [INFO] Lens API .......................................... SUCCESS [21.253s] > [INFO] Lens API for server and extensions ................ SUCCESS [20.861s] > [INFO] Lens Cube ......................................... SUCCESS [5:08.507s] > [INFO] Lens DB storage ................................... SUCCESS [19.065s] > [INFO] Lens Query Library ................................ SUCCESS [15.206s] > [INFO] Lens Hive Driver .................................. SUCCESS [2:50.257s] > [INFO] Lens Driver for JDBC .............................. SUCCESS [36.338s] > [INFO] Lens Elastic Search Driver ........................ SUCCESS [17.467s] > [INFO] Lens Server ....................................... SUCCESS [8:52.207s] > [INFO] Lens client ....................................... SUCCESS [38.217s] > [INFO] Lens CLI .......................................... SUCCESS [2:45.810s] > [INFO] Lens Examples ..................................... SUCCESS [10.357s] > [INFO] Lens Distribution ................................. SUCCESS [8.390s] > [INFO] Lens ML Lib ....................................... SUCCESS [1:22.195s] > [INFO] Lens ML Ext Distribution .......................... SUCCESS [2.081s] > [INFO] Lens Regression ................................... SUCCESS [11.032s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 24:25.067s > [INFO] Finished at: Tue Sep 22 14:20:49 UTC 2015 > [INFO] Final Memory: 186M/1378M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Rajat Khandelwal > >
