Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-3671: Add query option to limit scratch space usage

IMPALA-3671: Add query option to limit scratch space usage

Currently we can only disable spilling via a startup option which means
we need to restart the cluster for this.
This patch adds a new query option 'SCRATCH_LIMIT' that limits the amount of
scratch directory space that can be used. This would be useful to prevent
runaway queries or to prevent queries from spilling when that is not desired.
This also adds a 'ScratchSpace' counter to the runtime profile of the
BlockMgr that keeps track of the scratch space allocated.

Valid values for the SCRATCH_LIMIT query option are:
- unspecified or a limit of -1 means no limit
- a limit of 0 (zero) means spilling is disabled
- an int (= number of bytes)
- a float followed by "M" (MB) or "G" (GB)

A new test file "" was added for testing functionality.

Change-Id: Ibf8842626ded1345b632a0ccdb9a580e6a0ad470
Reviewed-by: Tim Armstrong <>
Tested-by: Internal Jenkins
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/buffered-block-mgr.h
M be/src/runtime/
M be/src/runtime/test-env.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/tmp-file-mgr.h
M be/src/service/
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/
A tests/query_test/
14 files changed, 407 insertions(+), 104 deletions(-)

  Internal Jenkins: Verified
  Tim Armstrong: Looks good to me, approved

To view, visit
To unsubscribe, visit

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibf8842626ded1345b632a0ccdb9a580e6a0ad470
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Bikramjeet Vig <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Tim Armstrong <>

Reply via email to