----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42067/#review113456 -----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 896) <https://reviews.apache.org/r/42067/#comment174178> Right. I wanted to do that initially, but due to latency that it will bring, I opted for driver initiating this for faster streaming. Also in server there are multiple flows that access the resultset which can create contention. Example the Result Formatter and the execute with timeout may start reading the result set parallay. The driver is the common place to manage this access and return a shared result set. lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 1806) <https://reviews.apache.org/r/42067/#comment174180> Hmm. Name should reflect what the method does. I saw in many places in code where rather than caching the context, getContext() was called on each subsequent line even though updated context was not required. Wanted to prompt developers that this method deos more than just getting the context. - Puneet Gupta On Jan. 8, 2016, 9:55 a.m., Puneet Gupta wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42067/ > ----------------------------------------------------------- > > (Updated Jan. 8, 2016, 9:55 a.m.) > > > Review request for lens. > > > Repository: lens > > > Description > ------- > > LENS-901 support streaming results on lens > > > Diffs > ----- > > > lens-api/src/main/java/org/apache/lens/api/query/QueryHandleWithResultSet.java > a5da867 > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 149c6ab > lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java > 82d7513 > > lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java > b96cf88 > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > 88e5a01 > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java > c64a3dd > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java > PRE-CREATION > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java > 1269e45 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > 2dff9af > lens-server/src/main/resources/lenssession-default.xml 52e0cd0 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java > c5d75de > > lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java > 30d1e34 > > Diff: https://reviews.apache.org/r/42067/diff/ > > > Testing > ------- > > Three properties have been added to enable streaming > > * lens.query.prefetch.inmemory.resultset > When set to true, specified number of rows of in-memory result set will be > pre-fetched > > * lens.query.prefetch.inmemory.resultset.rows > Specifies the number of rows to pre-fetch for in-memory result set when > lens.query.prefetch.inmemory.resultset is set to true. Default value is 100 > rows. > > * lens.query.prefetch.inmemory.resultset.ttl.millis Specifies the time in > milli seconds starting from query submission time for which the pre-fetched > in-memory result set will not be purged. This property is put to action only > when lens.query.prefetch.inmemory.resultset is set to true and all rows of > the in memory result set have been pre-fetched. The default value of this > property is 60 secs. > > Streaming(pre-fetched) result set is available in case the above proerties > are set and the result is a type of InMemoryResultSet. Further, the streming > is enabled only when all result rows have been prefetched.Partial Streaming > not supported as of now (Can be enabled in furtue is there is a clear use > case for it) > > Also note QueryHandleWithResultSet will now hold result metadta > (QueryResultSetMetadata) also. > This info will be useful for client in case the streaming results are > accesesd via execute_with_timeout API (REST End Point /queryapi/queries POST ) > > It is suggested to use this feature for queries that finish fast and return > small number of rows. Streaming and persistence by server can happen > parallely for streamed result sets. > > Will add more details on **Testing** done . Was able to build and run > checkstyle. Also ran the added test cases for this feature. > > **TODO** PartiallyFetchedInMemoryResultSet can override > InMemoryResultSet.toQueryResult() and optimize the case where result has been > completely fetched already. > > > Thanks, > > Puneet Gupta > >
