-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42067/
-----------------------------------------------------------
(Updated Feb. 3, 2016, 7:05 a.m.)
Review request for lens.
Changes
-------
- Updated Test Case to include all important methods of
PartiallyFetchedInMemoryResultSet .
- Removed seekToStart() form InMemoryResultSet ( since its not used and only
few drivers can support this feature. Can be added aback when we decide to
support the feature end to end)
- Removed unused member
PartiallyFetchedInMemoryResultSet.doNotPurgeUntilTimeMillis
Bugs: lens-901
https://issues.apache.org/jira/browse/lens-901
Repository: lens
Description
-------
support streaming results on lens
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/QueryHandleWithResultSet.java
a5da867
lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java 44fd97e
lens-driver-es/src/main/java/org/apache/lens/driver/es/client/ESResultSet.java
464b535
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
149c6ab
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveInMemoryResultSet.java
f8abd78
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java
82d7513
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCResultSet.java
8b4da3f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
b96cf88
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
8df389b
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
ed1fc43
lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java
f6434da
lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
1269e45
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java
a20cf47
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
e61398b
lens-server/src/main/resources/lenssession-default.xml a321c3f
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java
01e846a
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java
ef8c1aa
lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java
30d1e34
src/site/apt/admin/session-config.apt e108a13
Diff: https://reviews.apache.org/r/42067/diff/
Testing
-------
Three properties have been added to enable streaming
* lens.query.prefetch.inmemory.resultset
When set to true, specified number of rows of in-memory result set will be
pre-fetched
* lens.query.prefetch.inmemory.resultset.rows
Specifies the number of rows to pre-fetch for in-memory result set when
lens.query.prefetch.inmemory.resultset is set to true. Default value is 100
rows.
* lens.query.prefetch.inmemory.resultset.ttl.millis Specifies the time in milli
seconds starting from query submission time for which the pre-fetched in-memory
result set will not be purged. This property is put to action only when
lens.query.prefetch.inmemory.resultset is set to true and all rows of the in
memory result set have been pre-fetched. The default value of this property is
60 secs.
Streaming(pre-fetched) result set is available in case the above proerties are
set and the result is a type of InMemoryResultSet. Further, the streming is
enabled only when all result rows have been prefetched.Partial Streaming not
supported as of now (Can be enabled in furtue is there is a clear use case
for it)
Also note QueryHandleWithResultSet will now hold result metadta
(QueryResultSetMetadata) also.
This info will be useful for client in case the streaming results are accesesd
via execute_with_timeout API (REST End Point /queryapi/queries POST )
It is suggested to use this feature for queries that finish fast and return
small number of rows. Streaming and persistence by server can happen parallely
for streamed result sets.
Will add more details on **Testing** done . Was able to build and run
checkstyle. Also ran the added test cases for this feature.
**TODO** PartiallyFetchedInMemoryResultSet can override
InMemoryResultSet.toQueryResult() and optimize the case where result has been
completely fetched already.
Thanks,
Puneet Gupta
