----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42067/ -----------------------------------------------------------
Review request for lens. Repository: lens Description ------- LENS-901 support streaming results on lens Diffs ----- lens-api/src/main/java/org/apache/lens/api/query/QueryHandleWithResultSet.java a5da867 lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 149c6ab lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 82d7513 lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java b96cf88 lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 88e5a01 lens-server-api/src/main/java/org/apache/lens/server/api/driver/InMemoryResultSet.java c64a3dd lens-server-api/src/main/java/org/apache/lens/server/api/driver/PartiallyFetchedInMemoryResultSet.java PRE-CREATION lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 1269e45 lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 2dff9af lens-server/src/main/resources/lenssession-default.xml 52e0cd0 lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c5d75de lens-server/src/test/java/org/apache/lens/server/query/TestResultFormatting.java 30d1e34 Diff: https://reviews.apache.org/r/42067/diff/ Testing ------- Three properties have been added to enable streaming * lens.query.prefetch.inmemory.resultset When set to true, specified number of rows of in-memory result set will be pre-fetched * lens.query.prefetch.inmemory.resultset.rows Specifies the number of rows to pre-fetch for in-memory result set when lens.query.prefetch.inmemory.resultset is set to true. Default value is 100 rows. * lens.query.prefetch.inmemory.resultset.ttl.millis Specifies the time in milli seconds starting from query submission time for which the pre-fetched in-memory result set will not be purged. This property is put to action only when lens.query.prefetch.inmemory.resultset is set to true and all rows of the in memory result set have been pre-fetched. The default value of this property is 60 secs. Streaming(pre-fetched) result set is available in case the above proerties are set and the result is a type of InMemoryResultSet. Further, the streming is enabled only when all result rows have been prefetched.Partial Streaming not supported as of now (Can be enabled in furtue is there is a clear use case for it) Also note QueryHandleWithResultSet will now hold result metadta (QueryResultSetMetadata) also. This info will be useful for client in case the streaming results are accesesd via execute_with_timeout API (REST End Point /queryapi/queries POST ) It is suggested to use this feature for queries that finish fast and return small number of rows. Streaming and persistence by server can happen parallely for streamed result sets. Will add more details on **Testing** done . Was able to build and run checkstyle. Also ran the added test cases for this feature. **TODO** PartiallyFetchedInMemoryResultSet can override InMemoryResultSet.toQueryResult() and optimize the case where result has been completely fetched already. Thanks, Puneet Gupta
