-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37346/
-----------------------------------------------------------
(Updated Sept. 22, 2015, 8:16 p.m.)
Review request for lens.
Bugs: LENS-693
https://issues.apache.org/jira/browse/LENS-693
Repository: lens
Description
-------
Changes include:
1. The configuration `lens.server.max.finished.queries` has been taken off. Now
All queries are purged as soon as they are purgable. The logic of purgability
of a query is that:
1. if the query failed, it's purgable
2. if the query doesn't have a result, it's purgable
3. If it has a result, then result class is asked whether it's purgable or
not. This is the second layer of the logic. Current logic in that layer is as
follows:
1. If the result is persistent, it's purgable
2. If the result is in memory, it's not purgable initially, but after
all the rows have been accessed by someone, it becomes purgable. The rows can
be accessed in the following ways:
1. While sending Email notification, in memory result is appended
in the email. Hence rows are accessed.
2. Explicit call by a user for reading result set (/resultset REST
API, but not the /httpresultset API, http results are not available for in
memory results.)
2. There is a new configuration `lens.server.purge.interval`. The value is
supposed to be in milliseconds. This is the frequency of purger thread.
3. Purger thread had a bug in which it had a queue of finished queries which
were inserted according to their finish time. Purging of later queries was
dependent on purging of earlier queries. If a later query becomes purgable,
it's not purged untill all the queries before it become purgable and are
purged. To fix this, purger thread now runs periodically and scans the whole
queue, purges whichever ones are purgable.
4. FinishedLensQuery is a model object, but has a field which is not a field in
corresponding database. That field is removed from equals check
5. `instanceof` Checks have been converted to polymorphism:
1. For http result set availability.
2. For forming email message of the result of a finished query.
6. Email notifications are now sent irrespective of whether the query has been
purged or not. Earlier there was a bug that if a query is purged, email
notification for that isn't sent.
7. Taking care of LENS-783
8. Modularizing testing of email notifications
9. Made more util methods in RestAPITestUtil which became re-usable in server
tests like TestQueryService.
10. Added Test cases for query purging.
11. The way we're taking care of purge happening before event handlers are done
with context is by utilizing garbage collection. We pass context to the
QueryEnded event object. Now whenever the event happens, it has access to the
context. It has access even when purger has purged the query(since the object
will only go obsolete if no one has a handle to it). In purging, the context is
removed from various query collections, but context object is intact. Whenever
all event handlers for the QueryEnded event are done, both the event object and
the query context object will be garbage collected.
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/LensConf.java
3b1ad87ffe16b572437b92d90c793ea1881c2e0d
lens-api/src/main/java/org/apache/lens/api/query/InMemoryQueryResult.java
6a9f4558afc3a8430b43dcc65d73c813575a334d
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java
af439ff58002b06e2631585d35dfe9754ce84d4e
lens-api/src/main/java/org/apache/lens/api/query/PersistentQueryResult.java
a2e10c6560aaa551a15408b2937c3d495b51af5f
lens-api/src/main/java/org/apache/lens/api/query/QueryResult.java
3ecf0c229494dd82a531bc5a310e3b056a86d451
lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java
3c8531feb40276e076622b32f15c2f3d7ee782b2
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java
006eaedcd5cdf86b9aae6c4bbfcd92ba6c1df8ba
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
31c343a7b72c13ff9eb9b01038574297629e381b
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HivePersistentResultSet.java
00e1e53444ca76fa5096efc69fc4bdfdd16d116a
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java
78b3320977f9b46b2719ba80190fa77e87ecc729
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
096d26ee669c7e8c30df134ac334877311c1812e
lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java
3b761268645acf150d149e51f69521bf164c61ee
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensResultSet.java
929a3026da9d9789b64ab1f04fdd063db3443432
lens-server-api/src/main/java/org/apache/lens/server/api/driver/PersistentResultSet.java
bbde1705fec7fb946708b17d57eb5ad74fcb7cef
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java
d8c04dbdfdbf5309ef4ec6b07de27de33d2ff517
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryCancelled.java
bc0465c782a86d484747f0375135191bef0c49d1
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryClosed.java
ea8f70fc54b7c646e982a4de882eea92124b73d5
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
12de0a51b33a5ca09cd494f2ea842794b800f9e6
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryEnded.java
3e9474c642827e401a1e243ae25d328557f1adc0
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryFailed.java
40a1c0f1078c82f882eab60a0f0b20b501789da5
lens-server-api/src/main/java/org/apache/lens/server/api/query/QuerySuccess.java
c551dbab463dc7dd199f4fa1c6c56c956f56a704
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java
b28669e9070578e5121eb4b1c59e8e3ac7cb83de
lens-server/src/main/java/org/apache/lens/server/query/LensPersistentResult.java
b65a5f4ac661f060b51d5ee1c899dfd93aa8be90
lens-server/src/main/java/org/apache/lens/server/query/QueryEndNotifier.java
45ba7ac0300cc7b1d5d2ab723c4adf8d60801e11
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
9e27dd427cd573f57b06a3c8157c49f723b54caf
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionStatisticsGenerator.java
b57bc640bee018c2b7da560c2c4bcad5e228d41a
lens-server/src/main/java/org/apache/lens/server/query/ResultFormatter.java
af42eb06e2c030a7040b970e982cab782b2bb251
lens-server/src/main/resources/lensserver-default.xml
3c8fc09242bb3f873aa43630c9f386435f10c659
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java
05bd99f8819477c9560e231148b1e9ef7a8f061b
lens-server/src/test/java/org/apache/lens/server/common/RestAPITestUtil.java
0945598eed81515702402e9933b2592186209508
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java
b8012be1c30b544e51228c4b9b8a2a7d8281d3bc
lens-server/src/test/java/org/apache/lens/server/query/TestQueryEndEmailNotifier.java
82ce205c8d54c6665262ee3baf8c521203743eb5
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java
fa8d6eed0b98e9debe5e1c6f138bb7b104dff135
lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java
98a658e9c02a9da9d6708e4db1df161c6ea1a784
lens-server/src/test/resources/lens-site.xml
f728810867b677b6c7fba29229206eed1937e37f
Diff: https://reviews.apache.org/r/37346/diff/
Testing
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [1.844s]
[INFO] Lens .............................................. SUCCESS [3.076s]
[INFO] Lens API .......................................... SUCCESS [21.253s]
[INFO] Lens API for server and extensions ................ SUCCESS [20.861s]
[INFO] Lens Cube ......................................... SUCCESS [5:08.507s]
[INFO] Lens DB storage ................................... SUCCESS [19.065s]
[INFO] Lens Query Library ................................ SUCCESS [15.206s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:50.257s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [36.338s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [17.467s]
[INFO] Lens Server ....................................... SUCCESS [8:52.207s]
[INFO] Lens client ....................................... SUCCESS [38.217s]
[INFO] Lens CLI .......................................... SUCCESS [2:45.810s]
[INFO] Lens Examples ..................................... SUCCESS [10.357s]
[INFO] Lens Distribution ................................. SUCCESS [8.390s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.195s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [2.081s]
[INFO] Lens Regression ................................... SUCCESS [11.032s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:25.067s
[INFO] Finished at: Tue Sep 22 14:20:49 UTC 2015
[INFO] Final Memory: 186M/1378M
[INFO] ------------------------------------------------------------------------
Thanks,
Rajat Khandelwal
