Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14129 )
Change subject: IMPALA-8819: BufferedPlanRootSink should handle non-default fetch sizes ...................................................................... IMPALA-8819: BufferedPlanRootSink should handle non-default fetch sizes Adds support for non-default fetch sizes when result spooling is enabled (the default is to return BATCH_SIZE rows for each fetch request). When result spooling is disabled, Impala can only return up to BATCH_SIZE rows because it only buffers a single RowBatch at a time. When result spooling is enabled, each fetch request returns exactly the number of rows requested assuming there are that many rows left in the result set. There is also an upper limit on the fetch size to prevent the resulting QueryResultSet from getting too big. Unlike the behavior when result spooling is disabled, fetches do not break on RowBatch boundaries. For example, when result spooling is disabled, if the fetch size is 10 and the batch size is 15, the second fetch will return 5 rows. However, when result spooling is enabled the second fetch will return 10 rows (assuming there is another RowBatch to read). Testing: * Ran core tests * Added new tests to test_result_spooling.py * Added new tests to buffered-tuple-stream-test to validate writing to a BufferedTupleStream before releasing row batches with 'attach_on_read' set to true. Change-Id: I8dd4b397ab6457a4f85e635f239b2c67130fcce4 Reviewed-on: http://gerrit.cloudera.org:8080/14129 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/query-state.h M tests/query_test/test_result_spooling.py 5 files changed, 306 insertions(+), 56 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14129 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8dd4b397ab6457a4f85e635f239b2c67130fcce4 Gerrit-Change-Number: 14129 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Sahil Takiar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
