Repository: incubator-impala Updated Branches: refs/heads/master 75027c913 -> ed0aa66ee
IMPALA-3671: Add SCRATCH_LIMIT query option. Clarify comment about # of DataNodes. Change-Id: I662a59a59da8a170a2710d4a5245363ae1e3f754 Reviewed-on: http://gerrit.cloudera.org:8080/5651 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/6a2c9049 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/6a2c9049 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/6a2c9049 Branch: refs/heads/master Commit: 6a2c9049ffe1fc914508c880ab66eaae8bd5f07e Parents: 75027c9 Author: John Russell <[email protected]> Authored: Mon Jan 9 16:34:58 2017 -0800 Committer: Impala Public Jenkins <[email protected]> Committed: Wed Jan 11 23:38:00 2017 +0000 ---------------------------------------------------------------------- docs/impala.ditamap | 1 + docs/topics/impala_scratch_limit.xml | 100 ++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/6a2c9049/docs/impala.ditamap ---------------------------------------------------------------------- diff --git a/docs/impala.ditamap b/docs/impala.ditamap index fe4624c..380f25b 100644 --- a/docs/impala.ditamap +++ b/docs/impala.ditamap @@ -1027,6 +1027,7 @@ https://issues.cloudera.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=p <topicref rev="2.6.0" href="topics/impala_s3_skip_insert_staging.xml"/> <keydef href="topics/impala_s3_skip_insert_staging.xml" keys="s3_skip_insert_staging"/> <topicref rev="2.5.0" href="topics/impala_scan_node_codegen_threshold.xml"/> + <topicref rev="2.8.0 IMPALA-3671" href="topics/impala_scratch_limit.xml"/> <keydef href="topics/impala_scan_node_codegen_threshold.xml" keys="scan_node_codegen_threshold"/> <!-- This option is for internal use only and might go away without ever being documented. --> <!-- <topicref href="topics/impala_seq_compression_mode.xml"/> --> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/6a2c9049/docs/topics/impala_scratch_limit.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_scratch_limit.xml b/docs/topics/impala_scratch_limit.xml new file mode 100644 index 0000000..79ec987 --- /dev/null +++ b/docs/topics/impala_scratch_limit.xml @@ -0,0 +1,100 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd"> +<concept rev="IMPALA-3671" id="scratch_limit"> + + <title>SCRATCH_LIMIT Query Option</title> + <titlealts audience="PDF"><navtitle>SCRATCH_LIMIT</navtitle></titlealts> + <prolog> + <metadata> + <data name="Category" value="Impala"/> + <data name="Category" value="Impala Query Options"/> + <data name="Category" value="Memory"/> + <data name="Category" value="Developers"/> + <data name="Category" value="Data Analysts"/> + </metadata> + </prolog> + + <conbody> + + <p> + <indexterm audience="hidden">SCRATCH_LIMIT query option</indexterm> + Specifies the maximum amount of disk storage, in bytes, that any Impala query can consume + on any host using the <q>spill to disk</q> mechanism that handles queries that exceed + the memory limit. + </p> + + <p conref="../shared/impala_common.xml#common/syntax_blurb"/> + + <p> + Specify the size in bytes, or with a trailing <codeph>m</codeph> or <codeph>g</codeph> character to indicate + megabytes or gigabytes. For example: + </p> + +<!-- Examples adapted from impala_parquet_file_size.xml. + To do: demonstrate with queries that succeed / fail + based on the query option setting, and interaction + with MEM_LIMIT. --> +<codeblock>-- 128 megabytes. +set SCRATCH_LIMIT=134217728 + +-- 512 megabytes. +set SCRATCH_LIMIT=512m; + +-- 1 gigabyte. +set SCRATCH_LIMIT=1g; +</codeblock> + + <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/> + + <p> + A value of zero turns off the spill to disk feature for queries + in the current session, causing them to fail immediately if they + exceed the memory limit. + </p> + + <p> + The amount of memory used per host for a query is limited by the + <codeph>MEM_LIMIT</codeph> query option. + </p> + + <p> + The more Impala daemon hosts in the cluster, the less memory is used on each host, + and therefore also less scratch space is required for queries that + exceed the memory limit. + </p> + + <p> + <b>Type:</b> numeric, with optional unit specifier + </p> + + <p> + <b>Default:</b> -1 (amount of spill space is unlimited) + </p> + + <p conref="../shared/impala_common.xml#common/related_info"/> + + <p> + <xref href="impala_scalability.xml#spill_to_disk"/>, + <xref href="impala_mem_limit.xml#mem_limit"/> + </p> + + </conbody> +</concept>
