-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42067/
-----------------------------------------------------------
(Updated Jan. 22, 2016, 10:52 a.m.)
Review request for lens.
Changes
-------
Moved Logic to create the partially fetched result set out of specific driver,
so that new drivers can make use of this feature with minimal/no changes
. Added default implmentation for LensDriver.fetchResultSet() in
AbstractLensDriver which will fetch and also cache the resultset(on first
call).Some drivers were caching the result at driver level and some not. So
multiple calls to drivers (say from prsistance layer and from streaming layer)
that were not caching the result were resulting in new instance of result sets
that was creating problem
. Added QueryContext.registerDriverResult(LensResultSet) which will hold/cache
the driver result.
. The drivers no longer decide when to create a partially fetched result set.
For results to be streamed to client:
1. Driver Result should be of type InMemory as only such results can be
streamed fast
2. Query result should be server persistent. Only in this case, an early
streaming is required by client
that starts even before server level result persistence finishes.
3. For asynchronous query streaming result does not make sense
4. PREFETCH_INMEMORY_RESULTSET = true, implies client intent to get early
streamed result
5. rowsToPreFetch should be >0
Note: ESDriver is nmot update as of now . May be we can check if it makes sens
for this driver to stream results.
Testing Done
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [1.485s]
[INFO] Lens .............................................. SUCCESS [3.344s]
[INFO] Lens API .......................................... SUCCESS [18.349s]
[INFO] Lens API for server and extensions ................ SUCCESS [16.788s]
[INFO] Lens Cube ......................................... SUCCESS [10:47.072s]
[INFO] Lens DB storage ................................... SUCCESS [15.699s]
[INFO] Lens Query Library ................................ SUCCESS [11.212s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:23.925s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:15.973s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [14.494s]
[INFO] Lens Server ....................................... SUCCESS [10:17.601s]
[INFO] Lens client ....................................... SUCCESS [30.610s]
[INFO] Lens CLI .......................................... SUCCESS [46.161s]
[INFO] Lens Examples ..................................... SUCCESS [5.530s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.918s]
[INFO] Lens Distribution ................................. SUCCESS [7.827s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:11.793s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.771s]
[INFO] Lens Regression ................................... SUCCESS [10.109s]
[INFO] Lens UI ........................................... SUCCESS [29.048s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
Bugs: lens-901
https://issues.apache.org/jira/browse/lens-901
Repository: lens
Description
-------
LENS-901 support streaming results on lens
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/QueryHandleWithResultSet.java
a5da867
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
149c6ab
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java
82d7513
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
b96cf88
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
a3dbfc0
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
ed1fc43
lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java
c64a3dd
lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
1269e45
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
2dff9af
lens-server/src/main/resources/lenssession-default.xml a321c3f
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java
01e846a
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java
494bce5
lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java
30d1e34
src/site/apt/admin/config.apt 54f827e
src/site/apt/admin/session-config.apt 05a2c2c
Diff: https://reviews.apache.org/r/42067/diff/
Testing
-------
Three properties have been added to enable streaming
* lens.query.prefetch.inmemory.resultset
When set to true, specified number of rows of in-memory result set will be
pre-fetched
* lens.query.prefetch.inmemory.resultset.rows
Specifies the number of rows to pre-fetch for in-memory result set when
lens.query.prefetch.inmemory.resultset is set to true. Default value is 100
rows.
* lens.query.prefetch.inmemory.resultset.ttl.millis Specifies the time in milli
seconds starting from query submission time for which the pre-fetched in-memory
result set will not be purged. This property is put to action only when
lens.query.prefetch.inmemory.resultset is set to true and all rows of the in
memory result set have been pre-fetched. The default value of this property is
60 secs.
Streaming(pre-fetched) result set is available in case the above proerties are
set and the result is a type of InMemoryResultSet. Further, the streming is
enabled only when all result rows have been prefetched.Partial Streaming not
supported as of now (Can be enabled in furtue is there is a clear use case
for it)
Also note QueryHandleWithResultSet will now hold result metadta
(QueryResultSetMetadata) also.
This info will be useful for client in case the streaming results are accesesd
via execute_with_timeout API (REST End Point /queryapi/queries POST )
It is suggested to use this feature for queries that finish fast and return
small number of rows. Streaming and persistence by server can happen parallely
for streamed result sets.
Will add more details on **Testing** done . Was able to build and run
checkstyle. Also ran the added test cases for this feature.
**TODO** PartiallyFetchedInMemoryResultSet can override
InMemoryResultSet.toQueryResult() and optimize the case where result has been
completely fetched already.
Thanks,
Puneet Gupta
