> On Jan. 9, 2016, 2:48 a.m., Amareshwari Sriramadasu wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java,
> > lines 897-912
> > <https://reviews.apache.org/r/42067/diff/1/?file=1187330#file1187330line897>
> >
> > Shall we move this creation and caching to QueryContext itself sothat
> > each driver do not have a map and do not have same code to create
> > PartiallyFetchedInMemoryResultSet ?
* JDBC drivre was already caching query results in a Future object that is held
in the JdbcQueryContext. All results are cached here.
* Hive driver was not caching results, so a map was added for that to
HiveDriver Specifically to cache PartiallyFetchedInMemoryResultSet only since
this type of resultset is being accesed more than once. The cache is cleared on
query close().
The common code across drivers will have few lines of logic if caching is not
part of that logic.
public InMemoryResultSet
createPartiallyFetchedInMemoryResultSetifApplicable(InMemoryResultSet) {
if (pre fectch enabled and pre fetch rows > 0 ) {
return new PartiallyFetchedInMemoryResultSet (InMemoryResultSet , rows,
TTL)
}
}
1. Will update the QueryContext.isPreFetchInMemoryResultEnabled() to check both
enabled and rows > 0.
2. Should I add createPartiallyFetchedInMemoryResultSetifApplicable() method to
AbstractLensDriver. But still Hive Driverw ill have to maintain the cache.
> On Jan. 9, 2016, 2:48 a.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java,
> > line 89
> > <https://reviews.apache.org/r/42067/diff/1/?file=1187335#file1187335line89>
> >
> > Should we check if TTL has already expired?
Hmm, Can add this check. If TTL ahs already passed, we can stick to InMemory
Result only. The method createPartiallyFetchedInMemoryResultSetifApplicable()
can handle this check as well.
> On Jan. 9, 2016, 2:48 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java,
> > line 1958
> > <https://reviews.apache.org/r/42067/diff/1/?file=1187337#file1187337line1958>
> >
> > Should check for TTL ?
This check if not required I feel. This is because the TTL guarantess that
result will be available until TTL for sure, but it may be available till a few
seconds afterwards also until purging kicks in. Once Purged, the steraming will
not happen automatically.
> On Jan. 9, 2016, 2:48 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/resources/lenssession-default.xml, line 219
> > <https://reviews.apache.org/r/42067/diff/1/?file=1187338#file1187338line219>
> >
> > Can you update documentation about what happens if server persistence
> > is ON and prefetch inmemory reslt is enabled?
This propery is appicale to all InMemory result sets(irrespective of server
persistence). Should I mention how this property can be used ?
1. Server Persistence ON, Driver Persistence OFF.
Both persistence and streaming can happen from parallely.
2. Server Persistince OFF, Driver Persistence OFF.
> On Jan. 9, 2016, 2:48 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/resources/lenssession-default.xml, line 232
> > <https://reviews.apache.org/r/42067/diff/1/?file=1187338#file1187338line232>
> >
> > If query finishes after the TTL, what happens?
> >
> > If server persistence is ON and prefetch is enabled, and query finishes
> > after the TTL, what happens?
If the query finihses after TTL, it will not be prefetched . Check
org.apache.lens.server.api.query.QueryContext.registerDriverResult(LensResultSet)
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42067/#review113596
-----------------------------------------------------------
On Jan. 22, 2016, 10:52 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42067/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2016, 10:52 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: lens-901
> https://issues.apache.org/jira/browse/lens-901
>
>
> Repository: lens
>
>
> Description
> -------
>
> LENS-901 support streaming results on lens
>
>
> Diffs
> -----
>
>
> lens-api/src/main/java/org/apache/lens/api/query/QueryHandleWithResultSet.java
> a5da867
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> 149c6ab
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java
> 82d7513
>
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java
> b96cf88
>
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
> a3dbfc0
>
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
> ed1fc43
>
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java
> c64a3dd
>
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java
> PRE-CREATION
>
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
> 1269e45
>
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> 2dff9af
> lens-server/src/main/resources/lenssession-default.xml a321c3f
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java
> 01e846a
>
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java
> 494bce5
>
> lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java
> 30d1e34
> src/site/apt/admin/config.apt 54f827e
> src/site/apt/admin/session-config.apt 05a2c2c
>
> Diff: https://reviews.apache.org/r/42067/diff/
>
>
> Testing
> -------
>
> Three properties have been added to enable streaming
>
> * lens.query.prefetch.inmemory.resultset
> When set to true, specified number of rows of in-memory result set will be
> pre-fetched
>
> * lens.query.prefetch.inmemory.resultset.rows
> Specifies the number of rows to pre-fetch for in-memory result set when
> lens.query.prefetch.inmemory.resultset is set to true. Default value is 100
> rows.
>
> * lens.query.prefetch.inmemory.resultset.ttl.millis Specifies the time in
> milli seconds starting from query submission time for which the pre-fetched
> in-memory result set will not be purged. This property is put to action only
> when lens.query.prefetch.inmemory.resultset is set to true and all rows of
> the in memory result set have been pre-fetched. The default value of this
> property is 60 secs.
>
> Streaming(pre-fetched) result set is available in case the above proerties
> are set and the result is a type of InMemoryResultSet. Further, the streming
> is enabled only when all result rows have been prefetched.Partial Streaming
> not supported as of now (Can be enabled in furtue is there is a clear use
> case for it)
>
> Also note QueryHandleWithResultSet will now hold result metadta
> (QueryResultSetMetadata) also.
> This info will be useful for client in case the streaming results are
> accesesd via execute_with_timeout API (REST End Point /queryapi/queries POST )
>
> It is suggested to use this feature for queries that finish fast and return
> small number of rows. Streaming and persistence by server can happen
> parallely for streamed result sets.
>
> Will add more details on **Testing** done . Was able to build and run
> checkstyle. Also ran the added test cases for this feature.
>
> **TODO** PartiallyFetchedInMemoryResultSet can override
> InMemoryResultSet.toQueryResult() and optimize the case where result has been
> completely fetched already.
>
>
> Thanks,
>
> Puneet Gupta
>
>
