Repository: incubator-impala Updated Branches: refs/heads/master 749a55c4a -> 60127ff93
[DOCS] Release note updates for Impala 2.8 First cut at 'new features' topic. Includes the Incompatible Changes subtopic for Impala 2.8. Also did some cleanup throughout the Incompatible Changes page: - Took out references to Cloudera release numbers from titles. - Suppressed the display of ancient subtopics from the Impala beta days, which are intertwined with things like what version of Cloudera Manager was supported. Patch set 3: More on MT_DOP for COMPUTE STATS. Address comments from Greg and MJ. Added incompatible change for IMPALA-4160 - Llama removal. Also pulled out hidden discussion and obsolete details of no-op query options. Change-Id: I7c47f422e509cec6d3eb8aaa82294b584f393aed Reviewed-on: http://gerrit.cloudera.org:8080/5668 Reviewed-by: Jim Apple <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/60127ff9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/60127ff9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/60127ff9 Branch: refs/heads/master Commit: 60127ff93dee5d88a09f779f40c19f6d1dbf7a02 Parents: 749a55c Author: John Russell <[email protected]> Authored: Mon Jan 9 13:48:50 2017 -0800 Committer: Impala Public Jenkins <[email protected]> Committed: Thu Feb 23 18:02:47 2017 +0000 ---------------------------------------------------------------------- docs/shared/impala_common.xml | 18 +- docs/topics/impala_incompatible_changes.xml | 276 +++--- docs/topics/impala_new_features.xml | 877 +++++++++---------- .../impala_reservation_request_timeout.xml | 14 - docs/topics/impala_resource_management.xml | 118 +-- docs/topics/impala_v_cpu_cores.xml | 15 - 6 files changed, 523 insertions(+), 795 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/shared/impala_common.xml ---------------------------------------------------------------------- diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml index a21ff79..2d9656e 100644 --- a/docs/shared/impala_common.xml +++ b/docs/shared/impala_common.xml @@ -3290,19 +3290,18 @@ sudo pip-python install ssl</codeblock> <note id="llama_query_options_obsolete"> <p> - This query option is no longer supported, because it affects interaction - between Impala and Llama. - The use of the Llama component for integrated - resource management within YARN is no longer supported with CDH 5.5 / - Impala 2.3 and higher. + This query option no longer has any effect. + The use of the Llama component for integrated resource management within YARN is no + longer supported with <keyword keyref="impala23_full"/> and higher, and the Llama + support code is removed entirely in <keyword keyref="impala28_full"/> and higher. </p> </note> - <note id="impala_llama_obsolete"> + <note id="impala_llama_obsolete" rev="IMPALA-4160"> <p> - The use of the Llama component for integrated - resource management within YARN is no longer supported with CDH 5.5 / - Impala 2.3 and higher. + The use of the Llama component for integrated resource management within YARN + is no longer supported with <keyword keyref="impala23_full"/> and higher. + The Llama support code is removed entirely in <keyword keyref="impala28_full"/> and higher. </p> <p> For clusters running Impala alongside @@ -3319,7 +3318,6 @@ sudo pip-python install ssl</codeblock> Query Memory Limit</uicontrol> to produce a reliable estimate of overall memory consumption for a query. </note> - <p id="admission_control_mem_limit_interaction"> For example, consider the following scenario: <ul> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/topics/impala_incompatible_changes.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_incompatible_changes.xml b/docs/topics/impala_incompatible_changes.xml index b4cba7a..a25f327 100644 --- a/docs/topics/impala_incompatible_changes.xml +++ b/docs/topics/impala_incompatible_changes.xml @@ -53,9 +53,90 @@ under the License. <p outputclass="toc inpage"/> </conbody> + <concept rev="2.8.0" id="incompatible_changes_28x"> + + <title>Incompatible Changes Introduced in Impala 2.8.x</title> + + <conbody> + <ul> + <li> + <p rev="IMPALA-4160"> + Llama support is removed completely from Impala. Related flags (<codeph>--enable_rm</codeph>) + and query options (such as <codeph>V_CPU_CORES</codeph>) remain but do not have any effect. + </p> + <p rev="IMPALA-4160"> + If <codeph>--enable_rm</codeph> is passed to Impala, a warning is printed to the log on startup. + </p> + </li> + <li> + <p rev="kudu"> + The syntax related to Kudu tables includes a number of new reserved words, + such as <codeph>COMPRESSION</codeph>, <codeph>DEFAULT</codeph>, and <codeph>ENCODING</codeph>, that + might conflict with names of existing tables, columns, or other identifiers from older Impala versions. + See <xref href="impala_reserved_words.xml#reserved_words"/> for the full list of reserved words. + </p> + </li> + <li> + <p rev="kudu"> + The DDL syntax for Kudu tables, particularly in the <codeph>CREATE TABLE</codeph> statement, is different + from the special <codeph>impala_next</codeph> fork that was previously used for accessing Kudu tables + from Impala: + </p> + <ul rev="kudu"> + <li> + <p> + The <codeph>DISTRIBUTE BY</codeph> clause is now <codeph>PARTITIONED BY</codeph>. + </p> + </li> + <li> + <p> + The <codeph>INTO <varname>N</varname> BUCKETS</codeph> + clause is now <codeph>PARTITIONS <varname>N</varname></codeph>. + </p> + </li> + <li> + <p> + The <codeph>SPLIT ROWS</codeph> clause is replaced by different syntax for specifying + the ranges covered by each partition. + </p> + </li> + </ul> + </li> + <li> + <p> + The <codeph>DESCRIBE</codeph> output for Kudu tables includes several extra columns. + </p> + </li> + <li> + <p rev="kudu IMPALA-4527"> + Non-primary-key columns can contain <codeph>NULL</codeph> values by default. The + <codeph>SHOW CREATE TABLE</codeph> output for these columns displays the <codeph>NULL</codeph> + attribute. There was a period during early experimental versions of Impala + Kudu where + non-primary-key columns had the <codeph>NOT NULL</codeph> attribute by default. + </p> + </li> + <li> + <p rev="kudu IMPALA-3710"> + The <codeph>IGNORE</codeph> keyword that was present in early experimental versions of Impala + Kudu + is no longer present. The behavior of the <codeph>IGNORE</codeph> keyword is now the default: + DML statements continue with warnings, instead of failing with errors, if they encounter conditions + such as <q>primary key already exists</q> for an <codeph>INSERT</codeph> statement or + <q>primary key already deleted</q> for a <codeph>DELETE</codeph> statement. + </p> + </li> + <li> + <p rev="IMPALA-4589"> + The replication factor for Kudu tables must be an odd number. + </p> + </li> + </ul> + </conbody> + + </concept> + <concept rev="2.7.0" id="incompatible_changes_27x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.9.x / Impala 2.7.x</title> + <title>Incompatible Changes Introduced in Impala 2.7.x</title> <conbody> <ul> @@ -72,13 +153,6 @@ under the License. are now interpreted as <codeph>NULL</codeph> rather than being converted to valid floating-point values. </p> </li> - <li> - <p rev="IMPALA-4372"> - The column types shown in the <codeph>DESCRIBE FORMATTED</codeph> output are in uppercase, where formerly they - were in lowercase. This is not an intended change and could be reverted in the future, when IMPALA-4372 - is resolved. - </p> - </li> </ul> </conbody> @@ -86,13 +160,13 @@ under the License. <concept rev="2.6.0" id="incompatible_changes_26x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.8.x / Impala 2.6.x</title> + <title>Incompatible Changes Introduced in Impala 2.6.x</title> <conbody> <ul> <li> <p rev="CDH-41184"> - The default for the <codeph>RUNTIME_FILTER_MODE</codeph> + The default for the <codeph>RUNTIME_FILTER_MODE</codeph> query option is changed to <codeph>GLOBAL</codeph> (the highest setting). </p> </li> @@ -123,8 +197,8 @@ under the License. <p rev="IMPALA-3554"> For Kerberized clusters, the Catalog service now uses the Kerberos principal instead of the operating sytem user that runs - the <cmdname>catalogd</cmdname> daemon. - This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> + the <cmdname>catalogd</cmdname> daemon. + This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> setting to put the OS user into the Sentry administrative group, on clusters where the principal and the OS user name for this user are different. </p> @@ -218,7 +292,7 @@ under the License. <concept rev="2.5.x" id="incompatible_changes_25x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.7.x / Impala 2.5.x</title> + <title>Incompatible Changes Introduced in Impala 2.5.x</title> <conbody> <ul> @@ -306,7 +380,7 @@ under the License. <concept rev="2.4.x" id="incompatible_changes_24x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.6.x / Impala 2.4.x</title> + <title>Incompatible Changes Introduced in Impala 2.4.x</title> <conbody> <p> @@ -319,13 +393,13 @@ under the License. <!-- All 2.3.x subsections go under here --> -<!-- Actually for 2.3 / 5.5, let's get away from doing a separate subhead for each maintenance release, - because in the normal course of events there will be nothing to add here until 5.6. If something new - needs to get noted, just add a new bullet with wording to indicate which 5.5.x release it applies to. --> +<!-- Actually for 2.3 and higher, let's get away from doing a separate subhead for each maintenance release, + because in the normal course of events there will be nothing to add here until the next full release. If something new + needs to get noted, just add a new bullet with wording to indicate which x.y.z release it applies to. --> <concept rev="2.3.x" id="incompatible_changes_23x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.5.x / Impala 2.3.x</title> + <title>Incompatible Changes Introduced in Impala 2.3.x</title> <conbody> @@ -388,29 +462,10 @@ under the License. <concept rev="2.2.x" id="incompatible_changes_22x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.4.x</title> + <title>Incompatible Changes Introduced in Impala 2.2.x</title> <conbody> -<!-- - <p> - No incompatible changes. CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases. - See <xref href="impala_incompatible_changes.xml#incompatible_changes_220"/> for the initial Impala feature release, - which are the releases that typically include incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x"/> - - </conbody> - </concept> - - <concept rev="2.2.0" id="incompatible_changes_220"> - - <title>Incompatible Changes Introduced in Impala 2.2.0 / CDH 5.4.0</title> - - <conbody> ---> - <note conref="../shared/impala_common.xml#common/only_cdh5_220"/> <section id="files_220"> @@ -430,7 +485,6 @@ under the License. </p> </section> - <section id="prereqs_210"> <title> Changes to Prerequisites @@ -445,114 +499,10 @@ under the License. <concept rev="2.1.x" id="incompatible_changes_21x"> - <title>Incompatible Changes Introduced in Impala for CDH 5.3.x</title> + <title>Incompatible Changes Introduced in Impala 2.1.x</title> <conbody> -<!-- - <p> - No incompatible changes. CDH maintenance releases such as 5.3.1, 5.3.2, and so on are exclusively bug fix releases. - See <xref href="impala_incompatible_changes.xml#incompatible_changes_210"/> for the initial Impala feature release, - which are the releases that typically include incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - </concept> - - <concept rev="2.1.7" id="incompatible_changes_217"> - <title>Incompatible Changes Introduced in Cloudera Impala 2.1.7 / CDH 5.3.9</title> - <conbody> - <p> - No incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - </concept> - - <concept rev="2.1.6" id="incompatible_changes_216"> - <title>Incompatible Changes Introduced in Impala 2.1.6 / CDH 5.3.8</title> - <conbody> - <p> - No incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - </concept> - - <concept rev="2.1.5" id="incompatible_changes_215"> - <title>Incompatible Changes Introduced in Impala 2.1.5 / CDH 5.3.6</title> - <conbody> - <p> - No incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - </concept> - - <concept rev="2.1.4" id="incompatible_changes_214"> - <title>Incompatible Changes Introduced in Impala 2.1.4 / CDH 5.3.4</title> - <conbody> - <p> - No incompatible changes. - <ph conref="../shared/impala_common.xml#common/impala_214_redux"/> - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - </concept> - - <concept rev="2.1.3" id="incompatible_changes_213"> - <title>Incompatible Changes Introduced in Impala 2.1.3 / CDH 5.3.3</title> - <conbody> - <p> - No incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_213"/> - - </conbody> - </concept> - - <concept rev="2.1.2" id="incompatible_changes_212"> - <title>Incompatible Changes Introduced in Impala 2.1.2 / CDH 5.3.2</title> - <conbody> - <p> - No incompatible changes. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_212"/> - - </conbody> - </concept> - - <concept rev="2.1.1" id="incompatible_changes_211"> - - <title>Incompatible Changes Introduced in Impala 2.1.1 / CDH 5.3.1</title> - - <conbody> - - <p> - No incompatible changes. - </p> - </conbody> - </concept> - - <concept rev="2.1.0" id="incompatible_changes_210"> - - <title>Incompatible Changes Introduced in Impala 2.1.0 / CDH 5.3.0</title> - - <conbody> ---> - <section id="prereqs_210"> <title> Changes to Prerequisites @@ -594,7 +544,7 @@ under the License. <concept rev="2.0.5" id="incompatible_changes_205"> - <title>Incompatible Changes Introduced in Impala 2.0.5 / CDH 5.2.6</title> + <title>Incompatible Changes Introduced in Impala 2.0.5</title> <conbody> @@ -609,7 +559,7 @@ under the License. <concept rev="2.0.4" id="incompatible_changes_204"> - <title>Incompatible Changes Introduced in Impala 2.0.4 / CDH 5.2.5</title> + <title>Incompatible Changes Introduced in Impala 2.0.4</title> <conbody> @@ -624,7 +574,7 @@ under the License. <concept rev="2.0.3" id="incompatible_changes_203"> - <title>Incompatible Changes Introduced in Impala 2.0.3 / CDH 5.2.4</title> + <title>Incompatible Changes Introduced in Impala 2.0.3</title> <conbody> @@ -635,7 +585,7 @@ under the License. <concept rev="2.0.2" id="incompatible_changes_202"> - <title>Incompatible Changes Introduced in Impala 2.0.2 / CDH 5.2.3</title> + <title>Incompatible Changes Introduced in Impala 2.0.2</title> <conbody> @@ -650,7 +600,7 @@ under the License. <concept rev="2.0.1" id="incompatible_changes_201"> - <title>Incompatible Changes Introduced in Impala 2.0.1 / CDH 5.2.1</title> + <title>Incompatible Changes Introduced in Impala 2.0.1</title> <conbody> @@ -681,7 +631,7 @@ under the License. <concept rev="2.0.0" id="incompatible_changes_200"> - <title>Incompatible Changes Introduced in Impala 2.0.0 / CDH 5.2.0</title> + <title>Incompatible Changes Introduced in Impala 2.0.0</title> <conbody> @@ -873,7 +823,7 @@ under the License. </concept> <concept rev="1.4.4" id="incompatible_changes_144"> - <title>Incompatible Changes Introduced in Impala 1.4.4 / CDH 5.1.5</title> + <title>Incompatible Changes Introduced in Impala 1.4.4 /</title> <conbody> <p> No incompatible changes. @@ -886,7 +836,7 @@ under the License. <concept rev="1.4.3" id="incompatible_changes_143"> - <title>Incompatible Changes Introduced in Impala 1.4.3 / CDH 5.1.4</title> + <title>Incompatible Changes Introduced in Impala 1.4.3</title> <conbody> @@ -902,7 +852,7 @@ under the License. <concept rev="1.4.2" id="incompatible_changes_142"> - <title>Incompatible Changes Introduced in Impala 1.4.2 / CDH 5.1.3</title> + <title>Incompatible Changes Introduced in Impala 1.4.2</title> <conbody> @@ -917,7 +867,7 @@ under the License. <concept rev="1.4.1" id="incompatible_changes_141"> - <title>Incompatible Changes Introduced in Impala 1.4.1 / CDH 5.1.2</title> + <title>Incompatible Changes Introduced in Impala 1.4.1</title> <conbody> @@ -929,7 +879,7 @@ under the License. <concept rev="1.4.0" id="incompatible_changes_140"> - <title>Incompatible Changes Introduced in Impala 1.4.0 / CDH 5.1.0</title> + <title>Incompatible Changes Introduced in Impala 1.4.0</title> <prolog> <metadata> <data name="Category" value="Deprecated Features"/> @@ -1037,7 +987,7 @@ under the License. <concept rev="1.3.3" id="incompatible_changes_133"> - <title>Incompatible Changes Introduced in Impala 1.3.3 / CDH 5.0.5</title> + <title>Incompatible Changes Introduced in Impala 1.3.3</title> <conbody> @@ -1053,7 +1003,7 @@ under the License. <concept rev="1.3.2" id="incompatible_changes_132"> - <title>Incompatible Changes Introduced in Impala 1.3.2 / CDH 5.0.4</title> + <title>Incompatible Changes Introduced in Impala 1.3.2</title> <conbody> @@ -1068,7 +1018,7 @@ under the License. <concept rev="1.3.1" id="incompatible_changes_131"> - <title>Incompatible Changes Introduced in Impala 1.3.1 / CDH 5.0.3</title> + <title>Incompatible Changes Introduced in Impala 1.3.1</title> <conbody> @@ -1090,7 +1040,7 @@ under the License. <concept rev="1.3.0" id="incompatible_changes_130"> - <title>Incompatible Changes Introduced in Impala 1.3.0 / CDH 5.0.0</title> + <title>Incompatible Changes Introduced in Impala 1.3.0</title> <conbody> @@ -1467,7 +1417,7 @@ ALTER TABLE <varname>table_name</varname> SET FILEFORMAT </conbody> </concept> - <concept id="incompatible_changes_07"> + <concept id="incompatible_changes_07" audience="hidden"> <title>Incompatible Change Introduced in Version 0.7 of the Impala Beta Release</title> @@ -1484,7 +1434,7 @@ ALTER TABLE <varname>table_name</varname> SET FILEFORMAT </conbody> </concept> - <concept id="incompatible_changes_06"> + <concept id="incompatible_changes_06" audience="hidden"> <title>Incompatible Change Introduced in Version 0.6 of the Impala Beta Release</title> @@ -1501,7 +1451,7 @@ ALTER TABLE <varname>table_name</varname> SET FILEFORMAT </conbody> </concept> - <concept id="incompatible_changes_04"> + <concept id="incompatible_changes_04" audience="hidden"> <title>Incompatible Change Introduced in Version 0.4 of the Impala Beta Release</title> @@ -1518,7 +1468,7 @@ ALTER TABLE <varname>table_name</varname> SET FILEFORMAT </conbody> </concept> - <concept id="incompatible_changes_03"> + <concept id="incompatible_changes_03" audience="hidden"> <title>Incompatible Change Introduced in Version 0.3 of the Impala Beta Release</title> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/topics/impala_new_features.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_new_features.xml b/docs/topics/impala_new_features.xml index 5c8405d..4a237f7 100644 --- a/docs/topics/impala_new_features.xml +++ b/docs/topics/impala_new_features.xml @@ -46,11 +46,390 @@ under the License. </conbody> +<!-- All 2.8.x new features go under here --> + + <concept rev="2.8.0" id="new_features_280"> + + <title>New Features in <keyword keyref="impala28_full"/></title> + + <conbody> + + <ul id="feature_list"> + <li> + <p> + Performance and scalability improvements: + </p> + <ul> + <li> + <p rev="IMPALA-4572"> + The <codeph>COMPUTE STATS</codeph> statement can + take advantage of multithreading. + </p> + </li> + <li> + <p rev="IMPALA-4135"> + Improved scalability for highly concurrent loads by reducing the possibility of TCP/IP timeouts. + A configuration setting, <codeph>accepted_cnxn_queue_depth</codeph>, can be adjusted upwards to + avoid this type of timeout on large clusters. + </p> + </li> + <li> + <p> + Several performance improvements were made to the mechanism for generating native code: + </p> + <ul> + <li> + <p rev="IMPALA-3638"> + Some queries involving analytic functions can take better advantage of native code generation. + </p> + </li> + <li> + <p rev="IMPALA-4008"> + Modules produced during intermediate code generation are organized + to be easier to cache and reuse during the lifetime of a long-running or complicated query. + </p> + </li> + <li> + <p rev="IMPALA-4397 IMPALA-1430"> + The <codeph>COMPUTE STATS</codeph> statement is more efficient + (less time for the codegen phase) for tables with a large number + of columns, especially for tables containing <codeph>TIMESTAMP</codeph> + columns. + </p> + </li> + <li> + <p rev="IMPALA-3838 IMPALA-4495"> + The logic for determining whether or not to use a runtime filter is more reliable, and the + evaluation process itself is faster because of native code generation. + </p> + </li> + </ul> + </li> + <li> + <p rev="IMPALA-3902"> + The <codeph>MT_DOP</codeph> query option enables + multithreading for a number of Impala operations. + <codeph>COMPUTE STATS</codeph> statements for Parquet tables + use a default of <codeph>MT_DOP=4</codeph> to improve the + intra-node parallelism and CPU efficiency of this data-intensive + operation. + </p> + </li> + <li> + <p rev="IMPALA-4397"> + The <codeph>COMPUTE STATS</codeph> statement is more efficient + (less time for the codegen phase) for tables with a large number + of columns. + </p> + </li> + <li> + <p rev="IMPALA-2521"> + A new hint, <codeph>CLUSTERED</codeph>, + allows Impala <codeph>INSERT</codeph> operations on a Parquet table + that use dynamic partitioning to process a high number of + partitions in a single statement. The data is ordered based on the + partition key columns, and each partition is only written + by a single host, reducing the amount of memory needed to buffer + Parquet data while the data blocks are being constructed. + </p> + </li> + <li> + <p rev="IMPALA-4163"> + A new hint, <codeph>SORTBY(<varname>cols</varname>)</codeph>, + allows Impala <codeph>INSERT</codeph> operations on a Parquet table + to produce optimized output files with better compressibility + and a more compact range of min/max values within each data file. + </p> + </li> + <li> + <p rev="IMPALA-3552"> + The new configuration setting <codeph>inc_stats_size_limit_bytes</codeph> + lets you reduce the load on the catalog server when running the + <codeph>COMPUTE INCREMENTAL STATS</codeph> statement for very large tables. + </p> + </li> + <li> + <p rev="IMPALA-1788"> + Impala folds many constant expressions within query statements, + rather than evaluating them for each row. This optimization + is especially useful when using functions to manipulate and + format <codeph>TIMESTAMP</codeph> values, such as the result + of an expression such as <codeph>to_date(now() - interval 1 day)</codeph>. + </p> + </li> + <li> + <p rev="IMPALA-4529"> + Parsing of complicated expressions is faster. This speedup is + especially useful for queries containing large <codeph>CASE</codeph> + expressions. + </p> + </li> + <li> + <p rev="IMPALA-4302"> + Evaluation is faster for <codeph>IN</codeph> operators with many constant + arguments. The same performance improvement applies to other functions + with many constant arguments. + </p> + </li> + <li> + <p rev="IMPALA-1286"> + Impala optimizes identical comparison operators within multiple <codeph>OR</codeph> + blocks. + </p> + </li> + <li> + <p rev="IMPALA-4193 IMPALA-3342"> + The reporting for wall-clock times and total CPU time in profile output is more accurate. + </p> + </li> + <li> + <p rev="IMPALA-3671"> + A new query option, <codeph>SCRATCH_LIMIT</codeph>, lets you restrict the amount of + space used when a query exceeds the memory limit and activates the <q>spill to disk</q> mechanism. + This option helps to avoid runaway queries or make queries <q>fail fast</q> if they require more + memory than anticipated. You can prevent runaway queries from using excessive amounts of spill space, + without restarting the cluster to turn the spilling feature off entirely. + See <xref href="impala_scratch_limit.xml#scratch_limit"/> for details. + </p> + </li> + </ul> + </li> + <li> + <p> + Integration with Apache Kudu: + </p> + <ul> + <li> + <p rev=""> + The experimental Impala support for the Kudu storage layer has been folded + into the main Impala development branch. Impala can now directly access Kudu tables, + opening up new capabilities such as enhanced DML operations and continuous ingestion. + </p> + </li> + <li> + <p rev=""> + The <codeph>DELETE</codeph> statement is a flexible way to remove data from a Kudu table. Previously, + removing data from an Impala table involved removing or rewriting the underlying data files, dropping entire partitions, + or rewriting the entire table. This Impala statement only works for Kudu tables. + </p> + </li> + <li> + <p rev=""> + The <codeph>UPDATE</codeph> statement is a flexible way to modify data within a Kudu table. Previously, + updating data in an Impala table involved replacing the underlying data files, dropping entire partitions, + or rewriting the entire table. This Impala statement only works for Kudu tables. + </p> + </li> + <li> + <p rev="IMPALA-3725"> + The <codeph>UPSERT</codeph> statement is a flexible way to ingest, modify, or both data within a Kudu table. Previously, + ingesting data that might contain duplicates involved an inefficient multi-stage operation, and there was no + built-in protection against duplicate data. The <codeph>UPSERT</codeph> statement, in combination with + the primary key designation for Kudu tables, lets you add or replace rows in a single operation, and + automatically avoids creating any duplicate data. + </p> + </li> + <li> + <p rev="IMPALA-3719 IMPALA-3726"> + The <codeph>CREATE TABLE</codeph> statement gains some new clauses that are specific to Kudu tables: + <codeph>PARTITION BY</codeph>, <codeph>PARTITIONS</codeph>, <codeph>STORED AS KUDU</codeph>, and column + attributes <codeph>PRIMARY KEY</codeph>, <codeph>NULL</codeph> and <codeph>NOT NULL</codeph>, + <codeph>ENCODING</codeph>, <codeph>COMPRESSION</codeph>, <codeph>DEFAULT</codeph>, and <codeph>BLOCK_SIZE</codeph>. + These clauses replace the explicit <codeph>TBLPROPERTIES</codeph> settings that were required in the + early experimental phases of integration between Impala and Kudu. + </p> + </li> + <li> + <p rev="IMPALA-2890"> + The <codeph>ALTER TABLE</codeph> statement can change certain attributes of Kudu tables. + You can add, drop, or rename columns. + You can add or drop range partitions. + You can change the <codeph>TBLPROPERTIES</codeph> value to rename or point to a different underlying Kudu table, + independently from the Impala table name in the metastore database. + You cannot change the data type of an existing column in a Kudu table. + </p> + </li> + <li> + <p rev="IMPALA-4403"> + The <codeph>SHOW PARTITIONS</codeph> statement displays information about the distribution of data + between partitions in Kudu tables. A new variation, <codeph>SHOW RANGE PARTITIONS</codeph>, + displays information about the Kudu-specific partitions that apply across ranges of key values. + </p> + </li> + <li> + <p rev="IMPALA-4379"> + Not all Impala data types are supported in Kudu tables. In particular, currently the Impala + <codeph>TIMESTAMP</codeph> type is not allowed in a Kudu table. Impala does not recognize the + <codeph>UNIXTIME_MICROS</codeph> Kudu type when it is present in a Kudu table. (These two + representations of date/time data use different units and are not directly compatible.) + You cannot create columns of type <codeph>TIMESTAMP</codeph>, <codeph>DECIMAL</codeph>, + <codeph>VARCHAR</codeph>, or <codeph>CHAR</codeph> within a Kudu table. Within a query, you can + cast values in a result set to these types. Certain types, such as <codeph>BOOLEAN</codeph>, + cannot be used as primary key columns. + </p> + </li> + <li> + <p rev=""> + Currently, Kudu tables are not interchangeable between Impala and Hive the way other kinds of Impala tables are. + Although the metadata for Kudu tables is stored in the metastore database, currently Hive cannot access Kudu tables. + </p> + </li> + <li> + <p rev=""> + The <codeph>INSERT</codeph> statement works for Kudu tables. The organization + of the Kudu data makes it more efficient than with HDFS-backed tables to insert + data in small batches, such as with the <codeph>INSERT ... VALUES</codeph> syntax. + </p> + </li> + <li> + <p rev="IMPALA-4283"> + Some audit data is recorded for data governance purposes. + All <codeph>UPDATE</codeph>, <codeph>DELETE</codeph>, and <codeph>UPSERT</codeph> statements are characterized + as <codeph>INSERT</codeph> operations in the audit log. Currently, lineage metadata is not generated for + <codeph>UPDATE</codeph> and <codeph>DELETE</codeph> operations on Kudu tables. + </p> + </li> + <li> + <p rev="IMPALA-4000"> + Currently, Kudu tables have limited support for Sentry: + <ul> + <li> + <p> + Access to Kudu tables must be granted to roles as usual. + </p> + </li> + <li> + <p> + Currently, access to a Kudu table through Sentry is <q>all or nothing</q>. + You cannot enforce finer-grained permissions such as at the column level, + or permissions on certain operations such as <codeph>INSERT</codeph>. + </p> + </li> + <li> + <p> + Only users with <codeph>ALL</codeph> privileges on <codeph>SERVER</codeph> can create external Kudu tables. + </p> + </li> + </ul> + Because non-SQL APIs can access Kudu data without going through Sentry + authorization, currently the Sentry support is considered preliminary. + </p> + </li> + <li> + <p rev="IMPALA-4571"> + Equality and <codeph>IN</codeph> predicates in Impala queries are pushed to + Kudu and evaluated efficiently by the Kudu storage layer. + </p> + </li> + </ul> + </li> + <li> + <p rev=""> + <b>Security:</b> + </p> + <ul> + <li> + <p> + Impala can take advantage of the S3 encrypted credential + store, to avoid exposing the secret key when accessing + data stored on S3. + </p> + </li> + </ul> + </li> + <li> + <p rev=""> + The <codeph>REFRESH</codeph> statement now updates information about HDFS block locations. + Therefore, you can perform a fast and efficient <codeph>REFRESH</codeph> after doing an HDFS + rebalancing operation instead of the more expensive <codeph>INVALIDATE METADATA</codeph> statement. + </p> + </li> + <li> + <p rev="IMPALA-1654"> + [<xref keyref="IMPALA-1654">IMPALA-1654</xref>] + Several kinds of DDL operations + can now work on a range of partitions. The partitions can be specified + using operators such as <codeph><</codeph>, <codeph>>=</codeph>, and + <codeph>!=</codeph> rather than just an equality predicate applying to a single + partition. + This new feature extends the syntax of several clauses + of the <codeph>ALTER TABLE</codeph> statement + (<codeph>DROP PARTITION</codeph>, <codeph>SET [UN]CACHED</codeph>, + <codeph>SET FILEFORMAT | SERDEPROPERTIES | TBLPROPERTIES</codeph>), + the <codeph>SHOW FILES</codeph> statement, and the + <codeph>COMPUTE INCREMENTAL STATS</codeph> statement. + It does not apply to statements that are defined to only apply to a single + partition, such as <codeph>LOAD DATA</codeph>, <codeph>ALTER TABLE ... ADD PARTITION</codeph>, + <codeph>SET LOCATION</codeph>, and <codeph>INSERT</codeph> with a static + partitioning clause. + </p> + </li> + <li> + <p rev="IMPALA-3973"> + The <codeph>instr()</codeph> function has optional second and third arguments, representing + the character to position to begin searching for the substring, and the Nth occurrence + of the substring to find. + </p> + </li> + <li> + <p rev="IMPALA-3441 IMPALA-4387"> + Improved error handling for malformed Avro data. In particular, incorrect + precision or scale for <codeph>DECIMAL</codeph> types is now handled. + </p> + </li> + <li> + <p> + Impala debug web UI: + </p> + <ul> + <li> + <p rev="IMPALA-1169"> + In addition to <q>inflight</q> and <q>finished</q> queries, the web UI + now also includes a section for <q>queued</q> queries. + </p> + </li> + <li> + <p rev="IMPALA-4048"> + The <uicontrol>/sessions</uicontrol> tab now clarifies how many of the displayed + sections are active, and lets you sort by <uicontrol>Expired</uicontrol> status + to distinguish active sessions from expired ones. + </p> + </li> + </ul> + </li> + <li> + <p rev="IMPALA-4020"> + Improved stability when DDL operations such as <codeph>CREATE DATABASE</codeph> + or <codeph>DROP DATABASE</codeph> are run in Hive at the same time as an Impala + <codeph>INVALIDATE METADATA</codeph> statement. + </p> + </li> + <li> + <p rev="IMPALA-1616"> + The <q>out of memory</q> error report was made more user-friendly, with additional + diagnostic information to help identify the spot where the memory limit was exceeded. + </p> + </li> + <li> + <p rev="IMPALA-3983 IMPALA-3974"> + Improved disk space usage for Java-based UDFs. Temporary copies of the associated JAR + files are removed when no longer needed, so that they do not accumulate across restarts + of the <cmdname>catalogd</cmdname> daemon and potentially cause an out-of-space condition. + These temporary files are also created in the directory specified by the <codeph>local_library_dir</codeph> + configuration setting, so that the storage for these temporary files can be independent + from any capacity limits on the <filepath>/tmp</filepath> filesystem. + </p> + </li> + </ul> + + </conbody> + </concept> + <!-- All 2.7.x new features go under here --> <concept rev="2.7.0" id="new_features_270"> - <title>New Features in Impala 2.7.x / CDH 5.9.x</title> + <title>New Features in <keyword keyref="impala27_full"/></title> <conbody> @@ -171,27 +550,6 @@ under the License. column definitions in Avro tables. </p> </li> - <li audience="hidden"><!-- Patch didn't make it into in <keyword keyref="impala27_full"/> --> - <p rev="IMPALA-1654 CDH-43747"> - [<xref href="https://issues.cloudera.org/browse/IMPALA-1654"; scope="external" format="html">IMPALA-1654</xref>] - Several kinds of DDL operations - can now work on a range of partitions. The partitions can be specified - using operators such as <codeph><</codeph>, <codeph>>=</codeph>, and - <codeph>!=</codeph> rather than just an equality predicate applying to a single - partition. - This new feature extends the syntax of - several clauses - of the <codeph>ALTER TABLE</codeph> statement - (<codeph>DROP PARTITION</codeph>, <codeph>SET [UN]CACHED</codeph>, - <codeph>SET FILEFORMAT | SERDEPROPERTIES | TBLPROPERTIES</codeph>), - the <codeph>SHOW FILES</codeph> statement, and the - <codeph>COMPUTE INCREMENTAL STATS</codeph> statement. - It does not apply to statements that are defined to only apply to a single - partition, such as <codeph>LOAD DATA</codeph>, <codeph>ALTER TABLE ... ADD PARTITION</codeph>, - <codeph>SET LOCATION</codeph>, and <codeph>INSERT</codeph> with a static - partitioning clause. - </p> - </li> <li> <p rev="IMPALA-3575 CDH-43742"> [<xref href="https://issues.cloudera.org/browse/IMPALA-3575"; @@ -210,7 +568,7 @@ under the License. <concept rev="2.6.0" id="new_features_260"> - <title>New Features in Impala 2.6.x / CDH 5.8.x</title> + <title>New Features in <keyword keyref="impala26_full"/></title> <conbody> @@ -249,7 +607,7 @@ under the License. <ul> <li> <p rev="CDH-41184 IMPALA-3333"> - The default for the <codeph>RUNTIME_FILTER_MODE</codeph> + The default for the <codeph>RUNTIME_FILTER_MODE</codeph> query option is changed to <codeph>GLOBAL</codeph> (the highest setting). See <xref href="impala_runtime_filter_mode.xml#runtime_filter_mode"/> for details about this option. @@ -287,7 +645,7 @@ under the License. </li> <li rev="CDH-41184 IMPALA-3077"> <p> - Runtime filters can now be produced during join queries even + Runtime filters can now be produced during join queries even when the join processing activates the spill-to-disk mechanism. </p> </li> @@ -390,8 +748,8 @@ under the License. <p rev="IMPALA-3554"> For Kerberized clusters, the Catalog service now uses the Kerberos principal instead of the operating sytem user that runs - the <cmdname>catalogd</cmdname> daemon. - This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> + the <cmdname>catalogd</cmdname> daemon. + This eliminates the requirement to configure a <codeph>hadoop.user.group.static.mapping.overrides</codeph> setting to put the OS user into the Sentry administrative group, on clusters where the principal and the OS user name for this user are different. </p> @@ -543,7 +901,7 @@ under the License. <concept rev="2.5.0" id="new_features_250"> - <title>New Features in Impala 2.5.x / CDH 5.7.x</title> + <title>New Features in <keyword keyref="impala25_full"/></title> <conbody> @@ -899,7 +1257,7 @@ under the License. UDFs using new <codeph>CREATE FUNCTION</codeph> syntax, without a signature for arguments or the return value. Afterwards, you no longer have to run the <codeph>CREATE FUNCTION</codeph> statements again after a restart. - Although Impala does not have visibility into the UDFs that implement the + Although Impala does not have visibility into the UDFs that implement the Hive built-in functions, user-created Hive UDFs are now automatically available for calling through Impala. <ph audience="PDF">See <xref href="impala_create_function.xml#create_function"/> for details.</ph> @@ -978,7 +1336,7 @@ under the License. <concept rev="2.4.0" id="new_features_240"> - <title>New Features in Impala 2.4.x / CDH 5.6.x</title> + <title>New Features in <keyword keyref="impala24_full"/></title> <conbody> @@ -1004,7 +1362,7 @@ under the License. <concept rev="2.3.0" id="new_features_230"> - <title>New Features in Impala 2.3.x / CDH 5.5.x</title> + <title>New Features in <keyword keyref="impala23_full"/></title> <conbody> @@ -1377,106 +1735,6 @@ under the License. <!-- All 2.2.x subsections go under here --> -<!-- Removing all the 5.4.x release subtopics for which there wasn't anything to say. - Same convention as used in 5.5.x, 5.6.x, 5.7.x, 5.8.x. Only have one subtopic for - the .0. - <concept rev="5.4.10" id="new_features_2210"> - - <title>New Features in Impala 2.2.10 / CDH 5.4.10</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept rev="5.4.9" id="new_features_229"> - - <title>New Features in Impala 2.2.9 / CDH 5.4.9</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept rev="5.4.8" id="new_features_228"> - - <title>New Features in Impala 2.2.8 / CDH 5.4.8</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept rev="5.4.7" id="new_features_227"> - - <title>New Features in Impala 2.2.7 / CDH 5.4.7</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept audience="hidden" rev="5.4.6" id="new_features_226"> - - <title>New Features in Impala 2.2.6 / CDH 5.4.6</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept rev="5.4.5" id="new_features_225"> - - <title>New Features in Impala 2.2.x for CDH 5.4.5</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> ---> - <concept rev="5.4.3" id="new_features_223"> <title>New Features in Impala 2.2.x for CDH 5.4.3 and 5.4.4</title> @@ -1498,54 +1756,17 @@ under the License. </concept> -<!-- I let the 5.4.3/5.4.3 subtopic above remain in existence, but now back to hiding specific 5.4.x subtopics - after the .0 one that has the actual new features. - <concept audience="hidden" rev="5.4.2" id="new_features_222"> - - <title>New Features in Impala 2.2.x for CDH 5.4.2</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> - - <concept rev="5.4.x" id="new_features_54x"> - - <title>New Features in Impala for CDH 5.4.x</title> - - <conbody> - - <p> - See <xref href="impala_new_features.xml#new_features_220"/> for the most recent set of new Impala features. - CDH maintenance releases such as 5.4.1, 5.4.2, and so on are exclusively bug fix releases, - therefore there are no new features for the 5.4.x series after 5.4.0. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_22x" /> - - </conbody> - - </concept> ---> - <concept rev="2.2.0" id="new_features_220"> - <title>New Features in Impala 2.2.x / CDH 5.4.x</title> + <title>New Features in <keyword keyref="impala28_full"/></title> <conbody> <note conref="../shared/impala_common.xml#common/only_cdh5_220" /> <p> - The following are the major new features in Impala 2.2.x. This release, available as part of CDH - 5.4.x, contains improvements to performance, manageability, security, and SQL syntax. + The following are the major new features in <keyword keyref="impala22_full"/>. This release + contains improvements to performance, manageability, security, and SQL syntax. </p> <ul> @@ -1771,136 +1992,9 @@ under the License. <!-- All 2.1.x subsections go under here --> - <concept rev="2.1.8" id="new_features_218"> - - <title>New Features in Impala 2.1.8 / CDH 5.3.10</title> - - <conbody> - - <p> - This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - - </concept> - - <concept rev="2.1.7" id="new_features_217"> - - <title>New Features in Impala 2.1.7 / CDH 5.3.9</title> - - <conbody> - - <p> - This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - - </concept> - - <concept rev="2.1.6" id="new_features_216"> - - <title>New Features in Impala 2.1.6 / CDH 5.3.8</title> - - <conbody> - - <p> - This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - - </concept> - - <concept rev="2.1.5" id="new_features_215"> - - <title>New Features in Impala 2.1.5 / CDH 5.3.6</title> - - <conbody> - - <p> - This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - - </concept> - - <concept rev="2.1.4" id="new_features_214"> - - <title>New Features in Impala 2.1.4 / CDH 5.3.4</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - <ph conref="../shared/impala_common.xml#common/impala_214_redux"/> - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_21x"/> - - </conbody> - - </concept> - - <concept rev="2.1.3" id="new_features_213"> - - <title>New Features in Impala 2.1.3 / CDH 5.3.3</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_213" /> - - </conbody> - - </concept> - - <concept rev="2.1.2" id="new_features_212"> - - <title>New Features in Impala 2.1.2 / CDH 5.3.2</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_212" /> - - </conbody> - - </concept> - - <concept rev="2.1.1" id="new_features_211"> - - <title>New Features in Impala 2.1.1 / CDH 5.3.1</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - </conbody> - - </concept> - <concept rev="2.1.0" id="new_features_210"> - <title>New Features in Impala 2.1.0 / CDH 5.3.0</title> + <title>New Features in <keyword keyref="impala21_full"/></title> <conbody> @@ -1954,93 +2048,15 @@ under the License. <!-- All 2.0.x subsections go under here --> - <concept rev="2.0.5" id="new_features_205"> - - <title>New Features in Impala 2.0.5 / CDH 5.2.6</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_205"/> - - </conbody> - - </concept> - - <concept rev="2.0.4" id="new_features_204"> - - <title>New Features in Impala 2.0.4 / CDH 5.2.5</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_204" /> - - </conbody> - - </concept> - - <concept rev="2.0.3" id="new_features_203"> - - <title>New Features in Impala 2.0.3 / CDH 5.2.4</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_203" /> - - </conbody> - - </concept> - - <concept rev="2.0.2" id="new_features_202"> - - <title>New Features in Impala 2.0.2 / CDH 5.2.3</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_202" /> - - </conbody> - - </concept> - - <concept rev="2.0.1" id="new_features_201"> - - <title>New Features in Impala 2.0.1 / CDH 5.2.1</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - </conbody> - - </concept> - <concept rev="2.0.0" id="new_features_200"> - <title>New Features in Impala 2.0.0 / CDH 5.2.0</title> + <title>New Features in <keyword keyref="impala20_full"/></title> <conbody> <p> - The following are the major new features in Impala 2.0. This major release, available both with CDH 5.2 and - for CDH 4, contains improvements to performance, scalability, security, and SQL syntax. + The following are the major new features in <keyword keyref="impala20_full"/>. This major release + contains improvements to performance, scalability, security, and SQL syntax. </p> <ul> @@ -2364,75 +2380,14 @@ under the License. </concept> - <concept rev="1.4.4" id="new_features_144"> - <title>New Features in Impala 1.4.4 / CDH 5.1.5</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_144" /> - - </conbody> - - </concept> - - <concept rev="1.4.3" id="new_features_143"> - - <title>New Features in Impala 1.4.3 / CDH 5.1.4</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release for an SSL security issue. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_143" /> - - </conbody> - - </concept> - - <concept rev="1.4.2" id="new_features_142"> - - <title>New Features in Impala 1.4.2 / CDH 5.1.3</title> - - <conbody> - - <p> - Impala 1.4.2 is purely a bug-fix release. It does not include any new features. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_142" /> - - </conbody> - - </concept> - - <concept rev="1.4.1" id="new_features_141"> - - <title>New Features in Impala 1.4.1 / CDH 5.1.2</title> - - <conbody> - - <p> - Impala 1.4.1 is purely a bug-fix release. It does not include any new features. - </p> - - </conbody> - - </concept> - <concept rev="1.4.0" id="new_features_140"> - <title>New Features in Impala 1.4.0 / CDH 5.1.0</title> + <title>New Features in <keyword keyref="impala14_full"/></title> <conbody> <p> - The following are the major new features in Impala 1.4. + The following are the major new features in <keyword keyref="impala14_full"/>: </p> <ul> @@ -2685,22 +2640,6 @@ under the License. </concept> - <concept rev="1.3.3" id="new_features_133"> - - <title>New Features in Impala 1.3.3 / CDH 5.0.5</title> - - <conbody> - - <p> - No new features. This point release is exclusively a bug fix release for an SSL security issue. - </p> - - <note conref="../shared/impala_common.xml#common/only_cdh5_133" /> - - </conbody> - - </concept> - <concept rev="1.3.2" id="new_features_132"> <title>New Features in Impala 1.3.2 / CDH 5.0.4</title> @@ -2720,7 +2659,7 @@ under the License. <concept rev="1.3.1" id="new_features_131"> - <title>New Features in Impala 1.3.1 / CDH 5.0.3</title> + <title>New Features in Impala 1.3.1</title> <conbody> @@ -2729,11 +2668,6 @@ under the License. resulting from fixes for performance, reliability, or usability issues. </p> - <p> - Because 1.3.1 is the first 1.3.x release for CDH 4, if you are on CDH 4, also consult - <xref href="impala_new_features.xml#new_features_130"/> for more features that are new to you. - </p> - <note conref="../shared/impala_common.xml#common/impala13_cdh" /> <ul> @@ -2773,7 +2707,7 @@ under the License. <concept rev="1.3.0" id="new_features_130"> - <title>New Features in Impala 1.3.0 / CDH 5.0.0</title> + <title>New Features in <keyword keyref="impala13_full"/></title> <conbody> @@ -2904,7 +2838,7 @@ under the License. <conbody> <note> - Impala 1.2.4 works with CDH 4. It is primarily a bug fix release for Impala 1.2.3, plus some performance + Impala 1.2.4 is primarily a bug fix release for Impala 1.2.3, plus some performance enhancements for the catalog server to minimize startup and DDL wait times for Impala deployments with large numbers of databases, tables, and partitions. </note> @@ -2977,10 +2911,6 @@ under the License. <conbody> - <note> - Impala 1.2.3 works with CDH 4 and with CDH 5 beta 2. The resource management feature requires CDH 5 beta. - </note> - <p> Impala 1.2.3 contains exactly the same feature set as Impala 1.2.2. Its only difference is one additional fix for compatibility with Parquet files generated outside of Impala by components such as Hive, Pig, or @@ -2998,11 +2928,6 @@ under the License. <conbody> - <note> - Impala 1.2.2 works with CDH 4. Its feature set is a superset of features in the Impala 1.2.0 beta, with the - exception of resource management, which relies on CDH 5. - </note> - <p> Impala 1.2.2 includes new features for performance, security, and flexibility. The major enhancements over 1.2.1 are performance related, primarily for join queries. http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/topics/impala_reservation_request_timeout.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_reservation_request_timeout.xml b/docs/topics/impala_reservation_request_timeout.xml index 6f70cfd..61d111b 100644 --- a/docs/topics/impala_reservation_request_timeout.xml +++ b/docs/topics/impala_reservation_request_timeout.xml @@ -38,19 +38,5 @@ under the License. <note conref="../shared/impala_common.xml#common/llama_query_options_obsolete"/> - <p> - <indexterm audience="hidden">RESERVATION_REQUEST_TIMEOUT query option</indexterm> - Maximum number of milliseconds Impala will wait for a reservation to be completely granted or denied. Used in - conjunction with the Impala resource management feature in Impala 1.2 and higher with CDH 5. - </p> - - <p> - <b>Type:</b> numeric - </p> - - <p> - <b>Default:</b> 300000 (5 minutes) - </p> - </conbody> </concept> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/topics/impala_resource_management.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_resource_management.xml b/docs/topics/impala_resource_management.xml index bc98014..80dbbc4 100644 --- a/docs/topics/impala_resource_management.xml +++ b/docs/topics/impala_resource_management.xml @@ -44,113 +44,6 @@ under the License. <p outputclass="toc inpage"/> </conbody> - <concept audience="hidden" id="llama"> - <!-- Hiding the whole concept now that Llama is desupported. --> - - <title>The Llama Daemon</title> - <prolog> - <metadata> - <data name="Category" value="Llama"/> - </metadata> - </prolog> - - <conbody> - -<!-- Copied from http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_install_llama.html - turn into a conref in both places. --> - - <p> - Llama is a system that mediates resource management between Impala and Hadoop YARN. Llama enables - Impala to reserve, use, and release resource allocations in a Hadoop cluster. Llama is only required if - resource management is enabled in Impala. - </p> - - <p id="p_b2y_msl_jp"> - By default, YARN allocates resources bit-by-bit as needed by MapReduce jobs. Impala needs all resources - available at the same time, so that intermediate results can be exchanged between cluster nodes, and - queries do not stall partway through waiting for new resources to be allocated. Llama is the intermediary - process that ensures all requested resources are available before each Impala query actually begins. - </p> - -<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh_ig_llama_installation.html --> - -<!-- Highly likely to be removed per https://jira.cloudera.com/browse/CDH-21352, - so commenting out for now. Feels like there should be some 'installing llama via CM' topic - even if it says not to worry. - - <p> - For Llama installation instructions, see - <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_llama_installation.html"; scope="external" format="html">Llama - Installation</xref>. - </p> ---> - -<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Managing-Clusters/cm_mc_impala_service.html --> - - <p> - For management through Cloudera Manager, see - <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/admin_llama.html"; scope="external" format="html">The - Impala Llama ApplicationMaster</xref>. - </p> - </conbody> - </concept> - -<!-- Hiding more content per MJ feedback. - <concept id="rm_resource_estimates"> - - <title>Controlling Resource Estimation Behavior</title> - - <conbody> - - <p> - By default, Impala consults the table statistics and column statistics for each table in a query, and uses - those figures to construct estimates of needed resources for each query. See - <xref href="impala_compute_stats.xml#compute_stats"/> for the statement to collect table and column - statistics for a table. - </p> - - <p> - In <keyword keyref="impala25_full"/> and higher, the preferred way to avoid overcommitting memory in a high-concurrency - or multitenant scenario is to use Impala admission control together with dynamic resource pools. - You can specify a <uicontrol>Default Query Memory Limit</uicontrol> setting, with a different value for each - pool, and Impala uses that value to calculate how many queries can safely run within a specified - cluster-wide aggregate memory size. - See <xref href="impala_admission.xml#admission_control"/> for details. - </p> - - <p> - To avoid problems with inaccurate or missing statistics, which can lead to inaccurate estimates of resource - consumption, Impala allows you to set default estimates for CPU and memory resource consumption. - When the query is complete, those resources are returned to YARN as - normal. To enable this feature, use the command-line option <codeph>-rm_always_use_defaults</codeph> when - starting <cmdname>impalad</cmdname>, and optionally - <codeph>-rm_default_memory=<varname>size</varname></codeph> and <codeph>-rm_default_cpu_cores</codeph>. - See <xref href="impala_resource_management.xml#rm_options"/> for details about each option. - </p> - </conbody> - </concept> - - <concept id="rm_checking"> - - <title>Checking Resource Estimates and Actual Usage</title> - - <conbody> - - <p> - To make resource usage easier to verify, the output of the <codeph>EXPLAIN</codeph> SQL statement now - includes information about estimated memory usage, whether table and column statistics are available for - each table, and the number of virtual cores that a query will use. You can get this information through the - <codeph>EXPLAIN</codeph> statement without actually running the query. The extra information requires - setting the query option <codeph>EXPLAIN_LEVEL=verbose</codeph>; see - <xref href="impala_explain.xml#explain"/> for details. The same extended information is shown at the start - of the output from the <codeph>PROFILE</codeph> statement in <cmdname>impala-shell</cmdname>. The detailed - profile information is only available after running the query. You can take appropriate actions (gathering - statistics, adjusting query options) if you find that queries fail or run with suboptimal performance when - resource management is enabled. - </p> - </conbody> - </concept> ---> - <concept id="rm_enforcement"> <title>How Resource Limits Are Enforced</title> @@ -292,15 +185,6 @@ under the License. <xref href="impala_mem_limit.xml#mem_limit"/> </li> -<!-- Not supported in CDH 5.5 / Impala 2.3 and up. - <li> - <xref href="impala_reservation_request_timeout.xml#reservation_request_timeout"/> - </li> - - <li> - <xref href="impala_v_cpu_cores.xml#v_cpu_cores"/> - </li> ---> </ul> </conbody> </concept> @@ -319,7 +203,7 @@ under the License. inside the list, so not practical to enclose the whole thing in XML comments. --> <p audience="hidden"> - Currently, Impala in CDH 5 has the following limitations for resource management of Impala queries: + Currently, Impala has the following limitations for resource management of Impala queries: </p> <ul audience="hidden"> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/60127ff9/docs/topics/impala_v_cpu_cores.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_v_cpu_cores.xml b/docs/topics/impala_v_cpu_cores.xml index b2b68b1..5d1661a 100644 --- a/docs/topics/impala_v_cpu_cores.xml +++ b/docs/topics/impala_v_cpu_cores.xml @@ -38,20 +38,5 @@ under the License. <note conref="../shared/impala_common.xml#common/llama_query_options_obsolete"/> - <p> - <indexterm audience="hidden">V_CPU_CORES query option</indexterm> - The number of per-host virtual CPU cores to request from YARN. If set, the query option overrides the - automatic estimate from Impala. - Used in conjunction with the Impala resource management feature in Impala 1.2 and higher and CDH 5. - </p> - - <p> - <b>Type:</b> numeric - </p> - - <p> - <b>Default:</b> 0 (use automatic estimates) - </p> - </conbody> </concept>
