http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_hbase.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_hbase.html b/docs/build/html/topics/impala_hbase.html index 7ee8bad..7112f90 100644 --- a/docs/build/html/topics/impala_hbase.html +++ b/docs/build/html/topics/impala_hbase.html @@ -612,6 +612,15 @@ set hbase_caching=1000; The <code class="ph codeph">LOAD DATA</code> statement cannot be used with HBase tables. </p> </li> + <li class="li"> + <p class="p"> + The <code class="ph codeph">TABLESAMPLE</code> clause of the <code class="ph codeph">SELECT</code> + statement does not apply to a table reference derived from a view, a subquery, + or anything other than a real base table. This clause only works for tables + backed by HDFS or HDFS-like data files, therefore it does not apply to Kudu or + HBase tables. + </p> + </li> </ul> </div> </article>
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_hints.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_hints.html b/docs/build/html/topics/impala_hints.html index f2b48f0..19bd16d 100644 --- a/docs/build/html/topics/impala_hints.html +++ b/docs/build/html/topics/impala_hints.html @@ -111,11 +111,19 @@ INSERT <var class="keyword varname">insert_clauses</var> <p class="p"> With both forms of hint syntax, include the <code class="ph codeph">STRAIGHT_JOIN</code> - keyword immediately after the <code class="ph codeph">SELECT</code> keyword to prevent Impala from + keyword immediately after the <code class="ph codeph">SELECT</code> and any + <code class="ph codeph">DISTINCT</code> or <code class="ph codeph">ALL</code> keywords to prevent Impala from reordering the tables in a way that makes the join-related hints ineffective. </p> <p class="p"> + The <code class="ph codeph">STRAIGHT_JOIN</code> hint affects the join order of table references in the query + block containing the hint. It does not affect the join order of nested queries, such as views, + inline views, or <code class="ph codeph">WHERE</code>-clause subqueries. To use this hint for performance + tuning of complex queries, apply the hint to all query blocks that need a fixed join order. + </p> + + <p class="p"> To reduce the need to use hints, run the <code class="ph codeph">COMPUTE STATS</code> statement against all tables involved in joins, or used as the source tables for <code class="ph codeph">INSERT ... SELECT</code> operations where the destination is a partitioned Parquet table. Do this operation after loading data or making substantial http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_incompatible_changes.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_incompatible_changes.html b/docs/build/html/topics/impala_incompatible_changes.html index 9623ba1..d968e2b 100644 --- a/docs/build/html/topics/impala_incompatible_changes.html +++ b/docs/build/html/topics/impala_incompatible_changes.html @@ -23,9 +23,25 @@ <p class="p toc inpage"></p> </div> - <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_release_notes.html">Impala Release Notes</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="incompatible_changes__incompatible_changes_29x"> + <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_release_notes.html">Impala Release Notes</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="incompatible_changes__incompatible_changes_210x"> - <h2 class="title topictitle2" id="ariaid-title2">Incompatible Changes Introduced in Impala 2.9.x</h2> + <h2 class="title topictitle2" id="ariaid-title2">Incompatible Changes Introduced in Impala 2.10.x</h2> + + <div class="body conbody"> + + <p class="p"> + For the full list of issues closed in this release, including any that introduce + behavior changes or incompatibilities, see the + <a class="xref" href="https://impala.incubator.apache.org/docs/changelog-2.10.html"; target="_blank">changelog for <span class="keyword">Impala 2.10</span></a>. + </p> + + </div> + + </article> + + <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="incompatible_changes__incompatible_changes_29x"> + + <h2 class="title topictitle2" id="ariaid-title3">Incompatible Changes Introduced in Impala 2.9.x</h2> <div class="body conbody"> @@ -41,9 +57,9 @@ </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="incompatible_changes__incompatible_changes_28x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="incompatible_changes__incompatible_changes_28x"> - <h2 class="title topictitle2" id="ariaid-title3">Incompatible Changes Introduced in Impala 2.8.x</h2> + <h2 class="title topictitle2" id="ariaid-title4">Incompatible Changes Introduced in Impala 2.8.x</h2> <div class="body conbody"> <ul class="ul"> @@ -134,9 +150,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="incompatible_changes__incompatible_changes_27x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="incompatible_changes__incompatible_changes_27x"> - <h2 class="title topictitle2" id="ariaid-title4">Incompatible Changes Introduced in Impala 2.7.x</h2> + <h2 class="title topictitle2" id="ariaid-title5">Incompatible Changes Introduced in Impala 2.7.x</h2> <div class="body conbody"> <ul class="ul"> @@ -158,9 +174,9 @@ </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title5" id="incompatible_changes__incompatible_changes_26x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="incompatible_changes__incompatible_changes_26x"> - <h2 class="title topictitle2" id="ariaid-title5">Incompatible Changes Introduced in Impala 2.6.x</h2> + <h2 class="title topictitle2" id="ariaid-title6">Incompatible Changes Introduced in Impala 2.6.x</h2> <div class="body conbody"> <ul class="ul"> @@ -307,9 +323,9 @@ </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title6" id="incompatible_changes__incompatible_changes_25x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="incompatible_changes__incompatible_changes_25x"> - <h2 class="title topictitle2" id="ariaid-title6">Incompatible Changes Introduced in Impala 2.5.x</h2> + <h2 class="title topictitle2" id="ariaid-title7">Incompatible Changes Introduced in Impala 2.5.x</h2> <div class="body conbody"> <ul class="ul"> @@ -395,9 +411,9 @@ </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title7" id="incompatible_changes__incompatible_changes_24x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title8" id="incompatible_changes__incompatible_changes_24x"> - <h2 class="title topictitle2" id="ariaid-title7">Incompatible Changes Introduced in Impala 2.4.x</h2> + <h2 class="title topictitle2" id="ariaid-title8">Incompatible Changes Introduced in Impala 2.4.x</h2> <div class="body conbody"> <p class="p"> @@ -412,9 +428,9 @@ - <article class="topic concept nested1" aria-labelledby="ariaid-title8" id="incompatible_changes__incompatible_changes_23x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title9" id="incompatible_changes__incompatible_changes_23x"> - <h2 class="title topictitle2" id="ariaid-title8">Incompatible Changes Introduced in Impala 2.3.x</h2> + <h2 class="title topictitle2" id="ariaid-title9">Incompatible Changes Introduced in Impala 2.3.x</h2> <div class="body conbody"> @@ -482,9 +498,9 @@ - <article class="topic concept nested1" aria-labelledby="ariaid-title9" id="incompatible_changes__incompatible_changes_22x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title10" id="incompatible_changes__incompatible_changes_22x"> - <h2 class="title topictitle2" id="ariaid-title9">Incompatible Changes Introduced in Impala 2.2.x</h2> + <h2 class="title topictitle2" id="ariaid-title10">Incompatible Changes Introduced in Impala 2.2.x</h2> <div class="body conbody"> @@ -527,9 +543,9 @@ - <article class="topic concept nested1" aria-labelledby="ariaid-title10" id="incompatible_changes__incompatible_changes_21x"> + <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="incompatible_changes__incompatible_changes_21x"> - <h2 class="title topictitle2" id="ariaid-title10">Incompatible Changes Introduced in Impala 2.1.x</h2> + <h2 class="title topictitle2" id="ariaid-title11">Incompatible Changes Introduced in Impala 2.1.x</h2> <div class="body conbody"> @@ -572,9 +588,9 @@ - <article class="topic concept nested1" aria-labelledby="ariaid-title11" id="incompatible_changes__incompatible_changes_205"> + <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="incompatible_changes__incompatible_changes_205"> - <h2 class="title topictitle2" id="ariaid-title11">Incompatible Changes Introduced in Impala 2.0.5</h2> + <h2 class="title topictitle2" id="ariaid-title12">Incompatible Changes Introduced in Impala 2.0.5</h2> <div class="body conbody"> @@ -585,9 +601,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title12" id="incompatible_changes__incompatible_changes_204"> + <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="incompatible_changes__incompatible_changes_204"> - <h2 class="title topictitle2" id="ariaid-title12">Incompatible Changes Introduced in Impala 2.0.4</h2> + <h2 class="title topictitle2" id="ariaid-title13">Incompatible Changes Introduced in Impala 2.0.4</h2> <div class="body conbody"> @@ -598,18 +614,18 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title13" id="incompatible_changes__incompatible_changes_203"> + <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="incompatible_changes__incompatible_changes_203"> - <h2 class="title topictitle2" id="ariaid-title13">Incompatible Changes Introduced in Impala 2.0.3</h2> + <h2 class="title topictitle2" id="ariaid-title14">Incompatible Changes Introduced in Impala 2.0.3</h2> <div class="body conbody"> </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title14" id="incompatible_changes__incompatible_changes_202"> + <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="incompatible_changes__incompatible_changes_202"> - <h2 class="title topictitle2" id="ariaid-title14">Incompatible Changes Introduced in Impala 2.0.2</h2> + <h2 class="title topictitle2" id="ariaid-title15">Incompatible Changes Introduced in Impala 2.0.2</h2> <div class="body conbody"> @@ -620,9 +636,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title15" id="incompatible_changes__incompatible_changes_201"> + <article class="topic concept nested1" aria-labelledby="ariaid-title16" id="incompatible_changes__incompatible_changes_201"> - <h2 class="title topictitle2" id="ariaid-title15">Incompatible Changes Introduced in Impala 2.0.1</h2> + <h2 class="title topictitle2" id="ariaid-title16">Incompatible Changes Introduced in Impala 2.0.1</h2> <div class="body conbody"> @@ -659,9 +675,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title16" id="incompatible_changes__incompatible_changes_200"> + <article class="topic concept nested1" aria-labelledby="ariaid-title17" id="incompatible_changes__incompatible_changes_200"> - <h2 class="title topictitle2" id="ariaid-title16">Incompatible Changes Introduced in Impala 2.0.0</h2> + <h2 class="title topictitle2" id="ariaid-title17">Incompatible Changes Introduced in Impala 2.0.0</h2> <div class="body conbody"> @@ -855,8 +871,8 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title17" id="incompatible_changes__incompatible_changes_144"> - <h2 class="title topictitle2" id="ariaid-title17">Incompatible Changes Introduced in Impala 1.4.4</h2> + <article class="topic concept nested1" aria-labelledby="ariaid-title18" id="incompatible_changes__incompatible_changes_144"> + <h2 class="title topictitle2" id="ariaid-title18">Incompatible Changes Introduced in Impala 1.4.4</h2> <div class="body conbody"> <p class="p"> No incompatible changes. @@ -865,9 +881,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title18" id="incompatible_changes__incompatible_changes_143"> + <article class="topic concept nested1" aria-labelledby="ariaid-title19" id="incompatible_changes__incompatible_changes_143"> - <h2 class="title topictitle2" id="ariaid-title18">Incompatible Changes Introduced in Impala 1.4.3</h2> + <h2 class="title topictitle2" id="ariaid-title19">Incompatible Changes Introduced in Impala 1.4.3</h2> <div class="body conbody"> @@ -879,9 +895,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title19" id="incompatible_changes__incompatible_changes_142"> + <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="incompatible_changes__incompatible_changes_142"> - <h2 class="title topictitle2" id="ariaid-title19">Incompatible Changes Introduced in Impala 1.4.2</h2> + <h2 class="title topictitle2" id="ariaid-title20">Incompatible Changes Introduced in Impala 1.4.2</h2> <div class="body conbody"> @@ -892,9 +908,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title20" id="incompatible_changes__incompatible_changes_141"> + <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="incompatible_changes__incompatible_changes_141"> - <h2 class="title topictitle2" id="ariaid-title20">Incompatible Changes Introduced in Impala 1.4.1</h2> + <h2 class="title topictitle2" id="ariaid-title21">Incompatible Changes Introduced in Impala 1.4.1</h2> <div class="body conbody"> @@ -904,9 +920,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title21" id="incompatible_changes__incompatible_changes_140"> + <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="incompatible_changes__incompatible_changes_140"> - <h2 class="title topictitle2" id="ariaid-title21">Incompatible Changes Introduced in Impala 1.4.0</h2> + <h2 class="title topictitle2" id="ariaid-title22">Incompatible Changes Introduced in Impala 1.4.0</h2> <div class="body conbody"> @@ -1004,9 +1020,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title22" id="incompatible_changes__incompatible_changes_133"> + <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="incompatible_changes__incompatible_changes_133"> - <h2 class="title topictitle2" id="ariaid-title22">Incompatible Changes Introduced in Impala 1.3.3</h2> + <h2 class="title topictitle2" id="ariaid-title23">Incompatible Changes Introduced in Impala 1.3.3</h2> <div class="body conbody"> @@ -1018,9 +1034,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title23" id="incompatible_changes__incompatible_changes_132"> + <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="incompatible_changes__incompatible_changes_132"> - <h2 class="title topictitle2" id="ariaid-title23">Incompatible Changes Introduced in Impala 1.3.2</h2> + <h2 class="title topictitle2" id="ariaid-title24">Incompatible Changes Introduced in Impala 1.3.2</h2> <div class="body conbody"> @@ -1031,9 +1047,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title24" id="incompatible_changes__incompatible_changes_131"> + <article class="topic concept nested1" aria-labelledby="ariaid-title25" id="incompatible_changes__incompatible_changes_131"> - <h2 class="title topictitle2" id="ariaid-title24">Incompatible Changes Introduced in Impala 1.3.1</h2> + <h2 class="title topictitle2" id="ariaid-title25">Incompatible Changes Introduced in Impala 1.3.1</h2> <div class="body conbody"> @@ -1061,9 +1077,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title25" id="incompatible_changes__incompatible_changes_130"> + <article class="topic concept nested1" aria-labelledby="ariaid-title26" id="incompatible_changes__incompatible_changes_130"> - <h2 class="title topictitle2" id="ariaid-title25">Incompatible Changes Introduced in Impala 1.3.0</h2> + <h2 class="title topictitle2" id="ariaid-title26">Incompatible Changes Introduced in Impala 1.3.0</h2> <div class="body conbody"> @@ -1148,9 +1164,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title26" id="incompatible_changes__incompatible_changes_124"> + <article class="topic concept nested1" aria-labelledby="ariaid-title27" id="incompatible_changes__incompatible_changes_124"> - <h2 class="title topictitle2" id="ariaid-title26">Incompatible Changes Introduced in Impala 1.2.4</h2> + <h2 class="title topictitle2" id="ariaid-title27">Incompatible Changes Introduced in Impala 1.2.4</h2> <div class="body conbody"> @@ -1170,9 +1186,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title27" id="incompatible_changes__incompatible_changes_123"> + <article class="topic concept nested1" aria-labelledby="ariaid-title28" id="incompatible_changes__incompatible_changes_123"> - <h2 class="title topictitle2" id="ariaid-title27">Incompatible Changes Introduced in Impala 1.2.3</h2> + <h2 class="title topictitle2" id="ariaid-title28">Incompatible Changes Introduced in Impala 1.2.3</h2> <div class="body conbody"> @@ -1184,9 +1200,9 @@ </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title28" id="incompatible_changes__incompatible_changes_122"> + <article class="topic concept nested1" aria-labelledby="ariaid-title29" id="incompatible_changes__incompatible_changes_122"> - <h2 class="title topictitle2" id="ariaid-title28">Incompatible Changes Introduced in Impala 1.2.2</h2> + <h2 class="title topictitle2" id="ariaid-title29">Incompatible Changes Introduced in Impala 1.2.2</h2> <div class="body conbody"> @@ -1252,9 +1268,9 @@ select * from `cross`;</code></pre> </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title29" id="incompatible_changes__incompatible_changes_121"> + <article class="topic concept nested1" aria-labelledby="ariaid-title30" id="incompatible_changes__incompatible_changes_121"> - <h2 class="title topictitle2" id="ariaid-title29">Incompatible Changes Introduced in Impala 1.2.1</h2> + <h2 class="title topictitle2" id="ariaid-title30">Incompatible Changes Introduced in Impala 1.2.1</h2> <div class="body conbody"> @@ -1321,9 +1337,9 @@ select * from `cross`;</code></pre> </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title30" id="incompatible_changes__incompatible_changes_120"> + <article class="topic concept nested1" aria-labelledby="ariaid-title31" id="incompatible_changes__incompatible_changes_120"> - <h2 class="title topictitle2" id="ariaid-title30">Incompatible Changes Introduced in Impala 1.2.0 (Beta)</h2> + <h2 class="title topictitle2" id="ariaid-title31">Incompatible Changes Introduced in Impala 1.2.0 (Beta)</h2> <div class="body conbody"> @@ -1377,9 +1393,9 @@ select * from `cross`;</code></pre> </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title31" id="incompatible_changes__incompatible_changes_111"> + <article class="topic concept nested1" aria-labelledby="ariaid-title32" id="incompatible_changes__incompatible_changes_111"> - <h2 class="title topictitle2" id="ariaid-title31">Incompatible Changes Introduced in Impala 1.1.1</h2> + <h2 class="title topictitle2" id="ariaid-title32">Incompatible Changes Introduced in Impala 1.1.1</h2> <div class="body conbody"> @@ -1427,9 +1443,9 @@ ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title32" id="incompatible_changes__incompatible_changes_11"> + <article class="topic concept nested1" aria-labelledby="ariaid-title33" id="incompatible_changes__incompatible_changes_11"> - <h2 class="title topictitle2" id="ariaid-title32">Incompatible Change Introduced in Impala 1.1</h2> + <h2 class="title topictitle2" id="ariaid-title33">Incompatible Change Introduced in Impala 1.1</h2> <div class="body conbody"> @@ -1454,9 +1470,9 @@ ALTER TABLE <var class="keyword varname">table_name</var> SET FILEFORMAT </div> </article> - <article class="topic concept nested1" aria-labelledby="ariaid-title33" id="incompatible_changes__incompatible_changes_10"> + <article class="topic concept nested1" aria-labelledby="ariaid-title34" id="incompatible_changes__incompatible_changes_10"> - <h2 class="title topictitle2" id="ariaid-title33">Incompatible Changes Introduced in Impala 1.0</h2> + <h2 class="title topictitle2" id="ariaid-title34">Incompatible Changes Introduced in Impala 1.0</h2> <div class="body conbody"> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_joins.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_joins.html b/docs/build/html/topics/impala_joins.html index 436f9f5..b49dd3a 100644 --- a/docs/build/html/topics/impala_joins.html +++ b/docs/build/html/topics/impala_joins.html @@ -25,7 +25,7 @@ are supported in all Impala versions. The <code class="ph codeph">CROSS JOIN</code> operator is available in Impala 1.2.2 and higher. During performance tuning, you can override the reordering of join clauses that Impala does internally by including the keyword <code class="ph codeph">STRAIGHT_JOIN</code> immediately after the - <code class="ph codeph">SELECT</code> keyword + <code class="ph codeph">SELECT</code> and any <code class="ph codeph">DISTINCT</code> or <code class="ph codeph">ALL</code> keywords. </p> <pre class="pre codeblock"><code>SELECT <var class="keyword varname">select_list</var> FROM http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_known_issues.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_known_issues.html b/docs/build/html/topics/impala_known_issues.html index f9017d2..3b148ee 100644 --- a/docs/build/html/topics/impala_known_issues.html +++ b/docs/build/html/topics/impala_known_issues.html @@ -209,11 +209,37 @@ </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title11" id="known_issues_performance__IMPALA-1480"> + <article class="topic concept nested3" aria-labelledby="ariaid-title11" id="known_issues_performance__IMPALA-3316"> + <h4 class="title topictitle4" id="ariaid-title11">Slow queries for Parquet tables with convert_legacy_hive_parquet_utc_timestamps=true</h4> + <div class="body conbody"> + <p class="p"> + The configuration setting <code class="ph codeph">convert_legacy_hive_parquet_utc_timestamps=true</code> + uses an underlying function that can be a bottleneck on high volume, highly concurrent + queries due to the use of a global lock while loading time zone information. This bottleneck + can cause slowness when querying Parquet tables, up to 30x for scan-heavy queries. The amount + of slowdown depends on factors such as the number of cores and number of threads involved in the query. + </p> + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + The slowdown only occurs when accessing <code class="ph codeph">TIMESTAMP</code> columns within Parquet files that + were generated by Hive, and therefore require the on-the-fly timezone conversion processing. + </p> + </div> + <p class="p"><strong class="ph b">Bug:</strong> <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-3316"; target="_blank">IMPALA-3316</a></p> + <p class="p"><strong class="ph b">Severity:</strong> High</p> + <p class="p"><strong class="ph b">Workaround:</strong> If the <code class="ph codeph">TIMESTAMP</code> values stored in the table represent dates only, + with no time portion, consider storing them as strings in <code class="ph codeph">yyyy-MM-dd</code> format. + Impala implicitly converts such string values to <code class="ph codeph">TIMESTAMP</code> in calls to date/time + functions. + </p> + </div> + </article> + + <article class="topic concept nested3" aria-labelledby="ariaid-title12" id="known_issues_performance__IMPALA-1480"> - <h4 class="title topictitle4" id="ariaid-title11">Slow DDL statements for tables with large number of partitions</h4> + <h4 class="title topictitle4" id="ariaid-title12">Slow DDL statements for tables with large number of partitions</h4> <div class="body conbody"> @@ -235,6 +261,33 @@ </article> + <article class="topic concept nested3" aria-labelledby="ariaid-title13" id="known_issues_performance__ki_file_handle_cache"> + <h4 class="title topictitle4" id="ariaid-title13">Interaction of File Handle Cache with HDFS Appends and Short-Circuit Reads</h4> + <div class="body conbody"> + <p class="p"> + If a data file used by Impala is being continuously appended or overwritten in place by an + HDFS mechanism, such as <span class="keyword cmdname">hdfs dfs -appendToFile</span>, interaction with the + file handle caching feature in <span class="keyword">Impala 2.10</span> and higher could cause + short-circuit reads to sometimes be disabled on some DataNodes. When a mismatch is detected + between the cached file handle and a data block that was rewritten because of an append, + short-circuit reads are turned off on the affected host for a 10-minute period. + </p> + <p class="p"> + The possibility of encountering such an issue is the reason why the file handle caching + feature is currently turned off by default. See <a class="xref" href="impala_scalability.html">Scalability Considerations for Impala</a> + for information about this feature and how to enable it. + </p> + <p class="p"><strong class="ph b">Bug:</strong> <a class="xref" href="https://issues.apache.org/jira/browse/HDFS-12528"; target="_blank">HDFS-12528</a></p> + <p class="p"><strong class="ph b">Severity:</strong> High</p> + + <p class="p"><strong class="ph b">Workaround:</strong> Verify whether your ETL process is susceptible to this issue before enabling the file handle caching feature. + You can set the <span class="keyword cmdname">impalad</span> configuration option <code class="ph codeph">unused_file_handle_timeout_sec</code> to a time period + that is shorter than the HDFS setting <code class="ph codeph">dfs.client.read.shortcircuit.streams.cache.expiry.ms</code>. (Keep in mind that + the HDFS setting is in milliseconds while the Impala setting is in seconds.) + </p> + </div> + </article> + </article> <article class="topic concept nested2" aria-labelledby="known_issues_usability__ki_usability" id="known_issues_crash__known_issues_usability"> @@ -249,8 +302,8 @@ </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title13" id="known_issues_usability__IMPALA-4570"> - <h4 class="title topictitle4" id="ariaid-title13">Impala shell tarball is not usable on systems with setuptools versions where '0.7' is a substring of the full version string</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title15" id="known_issues_usability__IMPALA-4570"> + <h4 class="title topictitle4" id="ariaid-title15">Impala shell tarball is not usable on systems with setuptools versions where '0.7' is a substring of the full version string</h4> <div class="body conbody"> <p class="p"> For example, this issue could occur on a system using setuptools version 20.7.0. @@ -264,9 +317,9 @@ </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title14" id="known_issues_usability__IMPALA-3133"> + <article class="topic concept nested3" aria-labelledby="ariaid-title16" id="known_issues_usability__IMPALA-3133"> - <h4 class="title topictitle4" id="ariaid-title14">Unexpected privileges in show output</h4> + <h4 class="title topictitle4" id="ariaid-title16">Unexpected privileges in show output</h4> <div class="body conbody"> @@ -294,9 +347,9 @@ </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title15" id="known_issues_usability__IMPALA-1776"> + <article class="topic concept nested3" aria-labelledby="ariaid-title17" id="known_issues_usability__IMPALA-1776"> - <h4 class="title topictitle4" id="ariaid-title15">Less than 100% progress on completed simple SELECT queries</h4> + <h4 class="title topictitle4" id="ariaid-title17">Less than 100% progress on completed simple SELECT queries</h4> <div class="body conbody"> @@ -312,9 +365,9 @@ </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title16" id="known_issues_usability__concept_lmx_dk5_lx"> + <article class="topic concept nested3" aria-labelledby="ariaid-title18" id="known_issues_usability__concept_lmx_dk5_lx"> - <h4 class="title topictitle4" id="ariaid-title16">Unexpected column overflow behavior with INT datatypes</h4> + <h4 class="title topictitle4" id="ariaid-title18">Unexpected column overflow behavior with INT datatypes</h4> <div class="body conbody"> @@ -351,11 +404,11 @@ </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title18" id="known_issues_drivers__IMPALA-1792"> + <article class="topic concept nested3" aria-labelledby="ariaid-title20" id="known_issues_drivers__IMPALA-1792"> - <h4 class="title topictitle4" id="ariaid-title18">ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)</h4> + <h4 class="title topictitle4" id="ariaid-title20">ImpalaODBC: Can not get the value in the SQLGetData(m-x th column) after the SQLBindCol(m th column)</h4> <div class="body conbody"> @@ -392,11 +445,32 @@ </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title20" id="known_issues_security__renewable_kerberos_tickets"> + <article class="topic concept nested3" aria-labelledby="ariaid-title22" id="known_issues_security__IMPALA-5638"> + <h4 class="title topictitle4" id="ariaid-title22">Malicious user can gain unauthorized access to Kudu table data via Impala</h4> + <div class="body conbody"> + <p class="p"> + A malicious user with <code class="ph codeph">ALTER</code> permissions on an Impala table can access any + other Kudu table data by altering the table properties to make it <span class="q">"external"</span> + and then changing the underlying table mapping to point to other Kudu tables. + This violates and works around the authorization requirement that creating a + Kudu external table via Impala requires an <code class="ph codeph">ALL</code> privilege at the server scope. + This privilege requirement for <code class="ph codeph">CREATE</code> commands is enforced to precisely avoid + this scenario where a malicious user can change the underlying Kudu table + mapping. The fix is to enforce the same privilege requirement for <code class="ph codeph">ALTER</code> + commands that would make existing non-external Kudu tables external. + </p> + <p class="p"><strong class="ph b">Bug:</strong> <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-5638"; target="_blank">IMPALA-5638</a></p> + <p class="p"><strong class="ph b">Severity:</strong> High</p> + <p class="p"><strong class="ph b">Workaround:</strong> A temporary workaround is to revoke <code class="ph codeph">ALTER</code> permissions on Impala tables.</p> + <p class="p"><strong class="ph b">Resolution:</strong> Upgrade to an Impala version containing the fix for <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-5638"; target="_blank">IMPALA-5638</a>.</p> + </div> + </article> + + <article class="topic concept nested3" aria-labelledby="ariaid-title23" id="known_issues_security__renewable_kerberos_tickets"> - <h4 class="title topictitle4" id="ariaid-title20">Kerberos tickets must be renewable</h4> + <h4 class="title topictitle4" id="ariaid-title23">Kerberos tickets must be renewable</h4> <div class="body conbody"> @@ -432,8 +506,8 @@ </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title22" id="known_issues_resources__IMPALA-5605"> - <h4 class="title topictitle4" id="ariaid-title22">Configuration to prevent crashes caused by thread resource limits</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title25" id="known_issues_resources__IMPALA-5605"> + <h4 class="title topictitle4" id="ariaid-title25">Configuration to prevent crashes caused by thread resource limits</h4> <div class="body conbody"> <p class="p"> Impala could encounter a serious error due to resource usage under very high concurrency. @@ -465,8 +539,8 @@ impala hard nproc 262144 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title23" id="known_issues_resources__flatbuffers_mem_usage"> - <h4 class="title topictitle4" id="ariaid-title23">Memory usage when compact_catalog_topic flag enabled</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title26" id="known_issues_resources__flatbuffers_mem_usage"> + <h4 class="title topictitle4" id="ariaid-title26">Memory usage when compact_catalog_topic flag enabled</h4> <div class="body conbody"> <p class="p"> The efficiency improvement from <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-4029"; target="_blank">IMPALA-4029</a> @@ -491,8 +565,8 @@ impala hard nproc 262144 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title24" id="known_issues_resources__IMPALA-2294"> - <h4 class="title topictitle4" id="ariaid-title24">Kerberos initialization errors due to high memory usage</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title27" id="known_issues_resources__IMPALA-2294"> + <h4 class="title topictitle4" id="ariaid-title27">Kerberos initialization errors due to high memory usage</h4> <div class="body conbody"> <div class="p"> On a kerberized cluster with high memory utilization, <span class="keyword cmdname">kinit</span> commands executed after @@ -529,8 +603,8 @@ vm.overcommit_memory=1 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title25" id="known_issues_resources__drop_table_purge_s3a"> - <h4 class="title topictitle4" id="ariaid-title25">DROP TABLE PURGE on S3A table may not delete externally written files</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title28" id="known_issues_resources__drop_table_purge_s3a"> + <h4 class="title topictitle4" id="ariaid-title28">DROP TABLE PURGE on S3A table may not delete externally written files</h4> <div class="body conbody"> <p class="p"> A <code class="ph codeph">DROP TABLE PURGE</code> statement against an S3 table could leave the data files @@ -543,9 +617,9 @@ vm.overcommit_memory=1 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title26" id="known_issues_resources__catalogd_heap"> + <article class="topic concept nested3" aria-labelledby="ariaid-title29" id="known_issues_resources__catalogd_heap"> - <h4 class="title topictitle4" id="ariaid-title26">Impala catalogd heap issues when upgrading to <span class="keyword">Impala 2.5</span></h4> + <h4 class="title topictitle4" id="ariaid-title29">Impala catalogd heap issues when upgrading to <span class="keyword">Impala 2.5</span></h4> <div class="body conbody"> @@ -628,9 +702,9 @@ vm.overcommit_memory=1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title27" id="known_issues_resources__IMPALA-3509"> + <article class="topic concept nested3" aria-labelledby="ariaid-title30" id="known_issues_resources__IMPALA-3509"> - <h4 class="title topictitle4" id="ariaid-title27">Breakpad minidumps can be very large when the thread count is high</h4> + <h4 class="title topictitle4" id="ariaid-title30">Breakpad minidumps can be very large when the thread count is high</h4> <div class="body conbody"> @@ -659,9 +733,9 @@ vm.overcommit_memory=1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title28" id="known_issues_resources__IMPALA-3662"> + <article class="topic concept nested3" aria-labelledby="ariaid-title31" id="known_issues_resources__IMPALA-3662"> - <h4 class="title topictitle4" id="ariaid-title28">Parquet scanner memory increase after IMPALA-2736</h4> + <h4 class="title topictitle4" id="ariaid-title31">Parquet scanner memory increase after IMPALA-2736</h4> <div class="body conbody"> @@ -725,9 +799,9 @@ vm.overcommit_memory=1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title29" id="known_issues_resources__IMPALA-691"> + <article class="topic concept nested3" aria-labelledby="ariaid-title32" id="known_issues_resources__IMPALA-691"> - <h4 class="title topictitle4" id="ariaid-title29">Process mem limit does not account for the JVM's memory usage</h4> + <h4 class="title topictitle4" id="ariaid-title32">Process mem limit does not account for the JVM's memory usage</h4> @@ -751,11 +825,11 @@ vm.overcommit_memory=1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title30" id="known_issues_resources__IMPALA-2375"> + <article class="topic concept nested3" aria-labelledby="ariaid-title33" id="known_issues_resources__IMPALA-2375"> - <h4 class="title topictitle4" id="ariaid-title30">Fix issues with the legacy join and agg nodes using --enable_partitioned_hash_join=false and --enable_partitioned_aggregation=false</h4> + <h4 class="title topictitle4" id="ariaid-title33">Fix issues with the legacy join and agg nodes using --enable_partitioned_hash_join=false and --enable_partitioned_aggregation=false</h4> <div class="body conbody"> @@ -789,8 +863,8 @@ vm.overcommit_memory=1 </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title32" id="known_issues_correctness__IMPALA-4513"> - <h4 class="title topictitle4" id="ariaid-title32">ABS(n) where n is the lowest bound for the int types returns negative values</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title35" id="known_issues_correctness__IMPALA-4513"> + <h4 class="title topictitle4" id="ariaid-title35">ABS(n) where n is the lowest bound for the int types returns negative values</h4> <div class="body conbody"> <p class="p"> If the <code class="ph codeph">abs()</code> function evaluates a number that is right at the lower bound for @@ -806,8 +880,8 @@ vm.overcommit_memory=1 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title33" id="known_issues_correctness__IMPALA-4266"> - <h4 class="title topictitle4" id="ariaid-title33">Java udf expression returning string in group by can give incorrect results.</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title36" id="known_issues_correctness__IMPALA-4266"> + <h4 class="title topictitle4" id="ariaid-title36">Java udf expression returning string in group by can give incorrect results.</h4> <div class="body conbody"> <p class="p"> If the <code class="ph codeph">GROUP BY</code> clause included a call to a Java UDF that returned a string value, @@ -823,9 +897,9 @@ vm.overcommit_memory=1 </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title34" id="known_issues_correctness__IMPALA-3084"> + <article class="topic concept nested3" aria-labelledby="ariaid-title37" id="known_issues_correctness__IMPALA-3084"> - <h4 class="title topictitle4" id="ariaid-title34">Incorrect assignment of NULL checking predicate through an outer join of a nested collection.</h4> + <h4 class="title topictitle4" id="ariaid-title37">Incorrect assignment of NULL checking predicate through an outer join of a nested collection.</h4> <div class="body conbody"> @@ -849,9 +923,9 @@ vm.overcommit_memory=1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title35" id="known_issues_correctness__IMPALA-3094"> + <article class="topic concept nested3" aria-labelledby="ariaid-title38" id="known_issues_correctness__IMPALA-3094"> - <h4 class="title topictitle4" id="ariaid-title35">Incorrect result due to constant evaluation in query with outer join</h4> + <h4 class="title topictitle4" id="ariaid-title38">Incorrect result due to constant evaluation in query with outer join</h4> <div class="body conbody"> @@ -894,9 +968,9 @@ explain SELECT 1 FROM alltypestiny a1 </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title36" id="known_issues_correctness__IMPALA-3126"> + <article class="topic concept nested3" aria-labelledby="ariaid-title39" id="known_issues_correctness__IMPALA-3126"> - <h4 class="title topictitle4" id="ariaid-title36">Incorrect assignment of an inner join On-clause predicate through an outer join.</h4> + <h4 class="title topictitle4" id="ariaid-title39">Incorrect assignment of an inner join On-clause predicate through an outer join.</h4> <div class="body conbody"> @@ -1072,9 +1146,9 @@ where b.int_col = c.int_col </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title37" id="known_issues_correctness__IMPALA-3006"> + <article class="topic concept nested3" aria-labelledby="ariaid-title40" id="known_issues_correctness__IMPALA-3006"> - <h4 class="title topictitle4" id="ariaid-title37">Impala may use incorrect bit order with BIT_PACKED encoding</h4> + <h4 class="title topictitle4" id="ariaid-title40">Impala may use incorrect bit order with BIT_PACKED encoding</h4> <div class="body conbody"> @@ -1095,9 +1169,9 @@ where b.int_col = c.int_col </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title38" id="known_issues_correctness__IMPALA-3082"> + <article class="topic concept nested3" aria-labelledby="ariaid-title41" id="known_issues_correctness__IMPALA-3082"> - <h4 class="title topictitle4" id="ariaid-title38">BST between 1972 and 1995</h4> + <h4 class="title topictitle4" id="ariaid-title41">BST between 1972 and 1995</h4> <div class="body conbody"> @@ -1126,9 +1200,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title39" id="known_issues_correctness__IMPALA-1170"> + <article class="topic concept nested3" aria-labelledby="ariaid-title42" id="known_issues_correctness__IMPALA-1170"> - <h4 class="title topictitle4" id="ariaid-title39">parse_url() returns incorrect result if @ character in URL</h4> + <h4 class="title topictitle4" id="ariaid-title42">parse_url() returns incorrect result if @ character in URL</h4> <div class="body conbody"> @@ -1147,9 +1221,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title40" id="known_issues_correctness__IMPALA-2422"> + <article class="topic concept nested3" aria-labelledby="ariaid-title43" id="known_issues_correctness__IMPALA-2422"> - <h4 class="title topictitle4" id="ariaid-title40">% escaping does not work correctly when occurs at the end in a LIKE clause</h4> + <h4 class="title topictitle4" id="ariaid-title43">% escaping does not work correctly when occurs at the end in a LIKE clause</h4> <div class="body conbody"> @@ -1166,9 +1240,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title41" id="known_issues_correctness__IMPALA-397"> + <article class="topic concept nested3" aria-labelledby="ariaid-title44" id="known_issues_correctness__IMPALA-397"> - <h4 class="title topictitle4" id="ariaid-title41">ORDER BY rand() does not work.</h4> + <h4 class="title topictitle4" id="ariaid-title44">ORDER BY rand() does not work.</h4> <div class="body conbody"> @@ -1185,9 +1259,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title42" id="known_issues_correctness__IMPALA-2643"> + <article class="topic concept nested3" aria-labelledby="ariaid-title45" id="known_issues_correctness__IMPALA-2643"> - <h4 class="title topictitle4" id="ariaid-title42">Duplicated column in inline view causes dropping null slots during scan</h4> + <h4 class="title topictitle4" id="ariaid-title45">Duplicated column in inline view causes dropping null slots during scan</h4> <div class="body conbody"> @@ -1210,11 +1284,11 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title43" id="known_issues_correctness__IMPALA-1459"> + <article class="topic concept nested3" aria-labelledby="ariaid-title46" id="known_issues_correctness__IMPALA-1459"> - <h4 class="title topictitle4" id="ariaid-title43">Incorrect assignment of predicates through an outer join in an inline view.</h4> + <h4 class="title topictitle4" id="ariaid-title46">Incorrect assignment of predicates through an outer join in an inline view.</h4> <div class="body conbody"> @@ -1233,9 +1307,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title44" id="known_issues_correctness__IMPALA-2603"> + <article class="topic concept nested3" aria-labelledby="ariaid-title47" id="known_issues_correctness__IMPALA-2603"> - <h4 class="title topictitle4" id="ariaid-title44">Crash: impala::Coordinator::ValidateCollectionSlots</h4> + <h4 class="title topictitle4" id="ariaid-title47">Crash: impala::Coordinator::ValidateCollectionSlots</h4> <div class="body conbody"> @@ -1252,9 +1326,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title45" id="known_issues_correctness__IMPALA-2665"> + <article class="topic concept nested3" aria-labelledby="ariaid-title48" id="known_issues_correctness__IMPALA-2665"> - <h4 class="title topictitle4" id="ariaid-title45">Incorrect assignment of On-clause predicate inside inline view with an outer join.</h4> + <h4 class="title topictitle4" id="ariaid-title48">Incorrect assignment of On-clause predicate inside inline view with an outer join.</h4> <div class="body conbody"> @@ -1287,9 +1361,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title46" id="known_issues_correctness__IMPALA-2144"> + <article class="topic concept nested3" aria-labelledby="ariaid-title49" id="known_issues_correctness__IMPALA-2144"> - <h4 class="title topictitle4" id="ariaid-title46">Wrong assignment of having clause predicate across outer join</h4> + <h4 class="title topictitle4" id="ariaid-title49">Wrong assignment of having clause predicate across outer join</h4> <div class="body conbody"> @@ -1308,9 +1382,9 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title47" id="known_issues_correctness__IMPALA-2093"> + <article class="topic concept nested3" aria-labelledby="ariaid-title50" id="known_issues_correctness__IMPALA-2093"> - <h4 class="title topictitle4" id="ariaid-title47">Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate</h4> + <h4 class="title topictitle4" id="ariaid-title50">Wrong plan of NOT IN aggregate subquery when a constant is used in subquery predicate</h4> <div class="body conbody"> @@ -1344,9 +1418,9 @@ select </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title49" id="known_issues_metadata__IMPALA-2648"> + <article class="topic concept nested3" aria-labelledby="ariaid-title52" id="known_issues_metadata__IMPALA-2648"> - <h4 class="title topictitle4" id="ariaid-title49">Catalogd may crash when loading metadata for tables with many partitions, many columns and with incremental stats</h4> + <h4 class="title topictitle4" id="ariaid-title52">Catalogd may crash when loading metadata for tables with many partitions, many columns and with incremental stats</h4> <div class="body conbody"> @@ -1371,11 +1445,11 @@ select </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title50" id="known_issues_metadata__IMPALA-1420"> + <article class="topic concept nested3" aria-labelledby="ariaid-title53" id="known_issues_metadata__IMPALA-1420"> - <h4 class="title topictitle4" id="ariaid-title50">Can't update stats manually via alter table after upgrading to <span class="keyword">Impala 2.0</span></h4> + <h4 class="title topictitle4" id="ariaid-title53">Can't update stats manually via alter table after upgrading to <span class="keyword">Impala 2.0</span></h4> <div class="body conbody"> @@ -1420,9 +1494,9 @@ select - <article class="topic concept nested3" aria-labelledby="ariaid-title52" id="known_issues_interop__describe_formatted_avro"> + <article class="topic concept nested3" aria-labelledby="ariaid-title55" id="known_issues_interop__describe_formatted_avro"> - <h4 class="title topictitle4" id="ariaid-title52">DESCRIBE FORMATTED gives error on Avro table</h4> + <h4 class="title topictitle4" id="ariaid-title55">DESCRIBE FORMATTED gives error on Avro table</h4> <div class="body conbody"> @@ -1454,11 +1528,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title53" id="known_issues_interop__IMP-469"> + <article class="topic concept nested3" aria-labelledby="ariaid-title56" id="known_issues_interop__IMP-469"> - <h4 class="title topictitle4" id="ariaid-title53">Deviation from Hive behavior: Impala does not do implicit casts between string and numeric and boolean types.</h4> + <h4 class="title topictitle4" id="ariaid-title56">Deviation from Hive behavior: Impala does not do implicit casts between string and numeric and boolean types.</h4> <div class="body conbody"> @@ -1474,11 +1548,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title54" id="known_issues_interop__IMP-175"> + <article class="topic concept nested3" aria-labelledby="ariaid-title57" id="known_issues_interop__IMP-175"> - <h4 class="title topictitle4" id="ariaid-title54">Deviation from Hive behavior: Out of range values float/double values are returned as maximum allowed value of type (Hive returns NULL)</h4> + <h4 class="title topictitle4" id="ariaid-title57">Deviation from Hive behavior: Out of range values float/double values are returned as maximum allowed value of type (Hive returns NULL)</h4> <div class="body conbody"> @@ -1495,11 +1569,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title55" id="known_issues_interop__flume_writeformat_text"> + <article class="topic concept nested3" aria-labelledby="ariaid-title58" id="known_issues_interop__flume_writeformat_text"> - <h4 class="title topictitle4" id="ariaid-title55">Configuration needed for Flume to be compatible with Impala</h4> + <h4 class="title topictitle4" id="ariaid-title58">Configuration needed for Flume to be compatible with Impala</h4> <div class="body conbody"> @@ -1518,11 +1592,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title56" id="known_issues_interop__IMPALA-635"> + <article class="topic concept nested3" aria-labelledby="ariaid-title59" id="known_issues_interop__IMPALA-635"> - <h4 class="title topictitle4" id="ariaid-title56">Avro Scanner fails to parse some schemas</h4> + <h4 class="title topictitle4" id="ariaid-title59">Avro Scanner fails to parse some schemas</h4> <div class="body conbody"> @@ -1548,11 +1622,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title57" id="known_issues_interop__IMPALA-1024"> + <article class="topic concept nested3" aria-labelledby="ariaid-title60" id="known_issues_interop__IMPALA-1024"> - <h4 class="title topictitle4" id="ariaid-title57">Impala BE cannot parse Avro schema that contains a trailing semi-colon</h4> + <h4 class="title topictitle4" id="ariaid-title60">Impala BE cannot parse Avro schema that contains a trailing semi-colon</h4> <div class="body conbody"> @@ -1572,11 +1646,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title58" id="known_issues_interop__IMPALA-2154"> + <article class="topic concept nested3" aria-labelledby="ariaid-title61" id="known_issues_interop__IMPALA-2154"> - <h4 class="title topictitle4" id="ariaid-title58">Fix decompressor to allow parsing gzips with multiple streams</h4> + <h4 class="title topictitle4" id="ariaid-title61">Fix decompressor to allow parsing gzips with multiple streams</h4> <div class="body conbody"> @@ -1599,11 +1673,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title59" id="known_issues_interop__IMPALA-1578"> + <article class="topic concept nested3" aria-labelledby="ariaid-title62" id="known_issues_interop__IMPALA-1578"> - <h4 class="title topictitle4" id="ariaid-title59">Impala incorrectly handles text data when the new line character \n\r is split between different HDFS block</h4> + <h4 class="title topictitle4" id="ariaid-title62">Impala incorrectly handles text data when the new line character \n\r is split between different HDFS block</h4> <div class="body conbody"> @@ -1626,11 +1700,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title60" id="known_issues_interop__IMPALA-1862"> + <article class="topic concept nested3" aria-labelledby="ariaid-title63" id="known_issues_interop__IMPALA-1862"> - <h4 class="title topictitle4" id="ariaid-title60">Invalid bool value not reported as a scanner error</h4> + <h4 class="title topictitle4" id="ariaid-title63">Invalid bool value not reported as a scanner error</h4> <div class="body conbody"> @@ -1648,11 +1722,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title61" id="known_issues_interop__IMPALA-1652"> + <article class="topic concept nested3" aria-labelledby="ariaid-title64" id="known_issues_interop__IMPALA-1652"> - <h4 class="title topictitle4" id="ariaid-title61">Incorrect results with basic predicate on CHAR typed column.</h4> + <h4 class="title topictitle4" id="ariaid-title64">Incorrect results with basic predicate on CHAR typed column.</h4> <div class="body conbody"> @@ -1676,9 +1750,9 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested2" aria-labelledby="ariaid-title62" id="known_issues_crash__known_issues_limitations"> + <article class="topic concept nested2" aria-labelledby="ariaid-title65" id="known_issues_crash__known_issues_limitations"> - <h3 class="title topictitle3" id="ariaid-title62">Impala Known Issues: Limitations</h3> + <h3 class="title topictitle3" id="ariaid-title65">Impala Known Issues: Limitations</h3> <div class="body conbody"> @@ -1689,8 +1763,8 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title63" id="known_issues_limitations__IMPALA-4551"> - <h4 class="title topictitle4" id="ariaid-title63">Set limits on size of expression trees</h4> + <article class="topic concept nested3" aria-labelledby="ariaid-title66" id="known_issues_limitations__IMPALA-4551"> + <h4 class="title topictitle4" id="ariaid-title66">Set limits on size of expression trees</h4> <div class="body conbody"> <p class="p"> Very deeply nested expressions within queries can exceed internal Impala limits, @@ -1704,11 +1778,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </div> </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title64" id="known_issues_limitations__IMPALA-77"> + <article class="topic concept nested3" aria-labelledby="ariaid-title67" id="known_issues_limitations__IMPALA-77"> - <h4 class="title topictitle4" id="ariaid-title64">Impala does not support running on clusters with federated namespaces</h4> + <h4 class="title topictitle4" id="ariaid-title67">Impala does not support running on clusters with federated namespaces</h4> <div class="body conbody"> @@ -1735,9 +1809,9 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested2" aria-labelledby="ariaid-title65" id="known_issues_crash__known_issues_misc"> + <article class="topic concept nested2" aria-labelledby="ariaid-title68" id="known_issues_crash__known_issues_misc"> - <h3 class="title topictitle3" id="ariaid-title65">Impala Known Issues: Miscellaneous / Older Issues</h3> + <h3 class="title topictitle3" id="ariaid-title68">Impala Known Issues: Miscellaneous / Older Issues</h3> <div class="body conbody"> @@ -1747,11 +1821,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </div> - <article class="topic concept nested3" aria-labelledby="ariaid-title66" id="known_issues_misc__IMPALA-2005"> + <article class="topic concept nested3" aria-labelledby="ariaid-title69" id="known_issues_misc__IMPALA-2005"> - <h4 class="title topictitle4" id="ariaid-title66">A failed CTAS does not drop the table if the insert fails.</h4> + <h4 class="title topictitle4" id="ariaid-title69">A failed CTAS does not drop the table if the insert fails.</h4> <div class="body conbody"> @@ -1772,11 +1846,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title67" id="known_issues_misc__IMPALA-1821"> + <article class="topic concept nested3" aria-labelledby="ariaid-title70" id="known_issues_misc__IMPALA-1821"> - <h4 class="title topictitle4" id="ariaid-title67">Casting scenarios with invalid/inconsistent results</h4> + <h4 class="title topictitle4" id="ariaid-title70">Casting scenarios with invalid/inconsistent results</h4> <div class="body conbody"> @@ -1796,11 +1870,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title68" id="known_issues_misc__IMPALA-1619"> + <article class="topic concept nested3" aria-labelledby="ariaid-title71" id="known_issues_misc__IMPALA-1619"> - <h4 class="title topictitle4" id="ariaid-title68">Support individual memory allocations larger than 1 GB</h4> + <h4 class="title topictitle4" id="ariaid-title71">Support individual memory allocations larger than 1 GB</h4> <div class="body conbody"> @@ -1820,11 +1894,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title69" id="known_issues_misc__IMPALA-941"> + <article class="topic concept nested3" aria-labelledby="ariaid-title72" id="known_issues_misc__IMPALA-941"> - <h4 class="title topictitle4" id="ariaid-title69">Impala Parser issue when using fully qualified table names that start with a number.</h4> + <h4 class="title topictitle4" id="ariaid-title72">Impala Parser issue when using fully qualified table names that start with a number.</h4> <div class="body conbody"> @@ -1845,11 +1919,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title70" id="known_issues_misc__IMPALA-532"> + <article class="topic concept nested3" aria-labelledby="ariaid-title73" id="known_issues_misc__IMPALA-532"> - <h4 class="title topictitle4" id="ariaid-title70">Impala should tolerate bad locale settings</h4> + <h4 class="title topictitle4" id="ariaid-title73">Impala should tolerate bad locale settings</h4> <div class="body conbody"> @@ -1874,11 +1948,11 @@ ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='TRUE'); </article> - <article class="topic concept nested3" aria-labelledby="ariaid-title71" id="known_issues_misc__IMP-1203"> + <article class="topic concept nested3" aria-labelledby="ariaid-title74" id="known_issues_misc__IMP-1203"> - <h4 class="title topictitle4" id="ariaid-title71">Log Level 3 Not Recommended for Impala</h4> + <h4 class="title topictitle4" id="ariaid-title74">Log Level 3 Not Recommended for Impala</h4> <div class="body conbody"> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_kudu.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_kudu.html b/docs/build/html/topics/impala_kudu.html index 1ede7b6..d610595 100644 --- a/docs/build/html/topics/impala_kudu.html +++ b/docs/build/html/topics/impala_kudu.html @@ -1389,6 +1389,14 @@ kudu.table_name | impala::some_database.table_name_demo the predicate pushdown for a specific query against a Kudu table. </p> + <p class="p"> + The <code class="ph codeph">TABLESAMPLE</code> clause of the <code class="ph codeph">SELECT</code> + statement does not apply to a table reference derived from a view, a subquery, + or anything other than a real base table. This clause only works for tables + backed by HDFS or HDFS-like data files, therefore it does not apply to Kudu or + HBase tables. + </p> + http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_logging.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_logging.html b/docs/build/html/topics/impala_logging.html index 1882468..995c817 100644 --- a/docs/build/html/topics/impala_logging.html +++ b/docs/build/html/topics/impala_logging.html @@ -341,8 +341,7 @@ I0107 08:42:12.292706 14876 logging.cc:76] Flags (see also /varz are on debug we <p class="p"> For more information on how to configure GLOG, including how to set variable logging levels for different system components, see - <a class="xref" href="http://google-glog.googlecode.com/svn/trunk/doc/glog.html"; target="_blank">How - To Use Google Logging Library (glog)</a>. + <a class="xref" href="https://github.com/google/glog"; target="_blank">documentation for the glog project on github</a>. </p> <section class="section" id="log_levels__loglevels_details"><h3 class="title sectiontitle">Understanding What is Logged at Different Logging Levels</h3> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_map.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_map.html b/docs/build/html/topics/impala_map.html index 96cd2e8..17a1c25 100644 --- a/docs/build/html/topics/impala_map.html +++ b/docs/build/html/topics/impala_map.html @@ -102,7 +102,7 @@ type ::= <var class="keyword varname">primitive_type</var> | <var class="keyword </p> </li> <li class="li"> - <p class="p" id="map__d6e3003"> + <p class="p" id="map__d6e3156"> The maximum length of the column definition for any complex type, including declarations for any nested types, is 4000 characters. </p> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_math_functions.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_math_functions.html b/docs/build/html/topics/impala_math_functions.html index 431e6c3..ce10a98 100644 --- a/docs/build/html/topics/impala_math_functions.html +++ b/docs/build/html/topics/impala_math_functions.html @@ -1413,66 +1413,209 @@ select x, unix_timestamp(now()), rand(unix_timestamp(now())) <dt class="dt dlterm" id="math_functions__truncate"> <code class="ph codeph">truncate(double_or_decimal a[, digits_to_leave])</code>, - <span class="ph" id="math_functions__dtrunc"><code class="ph codeph">dtrunc(double_or_decimal a[, digits_to_leave])</code></span> + <span class="ph" id="math_functions__dtrunc"><code class="ph codeph">dtrunc(double_or_decimal a[, digits_to_leave])</code></span>, + <span class="ph" id="math_functions__trunc_number"><code class="ph codeph">trunc(double_or_decimal a[, digits_to_leave])</code></span> </dt> <dd class="dd"> + <strong class="ph b">Purpose:</strong> Removes some or all fractional digits from a numeric value. - With no argument, removes all fractional digits, leaving an integer value. - The optional argument specifies the number of fractional digits to include - in the return value, and only applies with the argument type is <code class="ph codeph">DECIMAL</code>. - <code class="ph codeph">truncate()</code> and <code class="ph codeph">dtrunc()</code> are aliases for the same function. + <p class="p"> + <strong class="ph b">Arguments:</strong> + With a single floating-point argument, removes all fractional digits, leaving an + integer value. The optional second argument specifies the number of fractional digits + to include in the return value, and only applies when the argument type is + <code class="ph codeph">DECIMAL</code>. A second argument of 0 truncates to a whole integer value. + A second argument of negative N sets N digits to 0 on the left side of the decimal + </p> + <p class="p"> + <strong class="ph b">Scale argument:</strong> The scale argument applies only when truncating + <code class="ph codeph">DECIMAL</code> values. It is an integer specifying how many + significant digits to leave to the right of the decimal point. + A scale argument of 0 truncates to a whole integer value. A scale + argument of negative N sets N digits to 0 on the left side of the decimal + point. + </p> + <p class="p"> + <code class="ph codeph">truncate()</code>, <code class="ph codeph">dtrunc()</code>, + <span class="ph">and <code class="ph codeph">trunc()</code></span> are aliases for the + same function. + </p> <p class="p"> <strong class="ph b">Return type:</strong> <code class="ph codeph">decimal</code> for <code class="ph codeph">DECIMAL</code> arguments; <code class="ph codeph">bigint</code> for <code class="ph codeph">DOUBLE</code> arguments </p> <p class="p"> + <strong class="ph b">Added in:</strong> The <code class="ph codeph">trunc()</code> alias was added in + <span class="keyword">Impala 2.10</span>. + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + <p class="p"> + You can also pass a <code class="ph codeph">DOUBLE</code> argument, or <code class="ph codeph">DECIMAL</code> + argument with optional scale, to the <code class="ph codeph">dtrunc()</code> or + <code class="ph codeph">truncate</code> functions. Using the <code class="ph codeph">trunc()</code> + function for numeric values is common with other industry-standard database + systems, so you might find such <code class="ph codeph">trunc()</code> calls in code that you + are porting to Impala. + </p> + <p class="p"> + The <code class="ph codeph">trunc()</code> function also has a signature that applies to + <code class="ph codeph">TIMESTAMP</code> values. See <a class="xref" href="impala_datetime_functions.html">Impala Date and Time Functions</a> + for details. + </p> + <p class="p"> <strong class="ph b">Examples:</strong> </p> -<pre class="pre codeblock"><code>select truncate(3.45) + <p class="p"> + The following examples demonstrate the <code class="ph codeph">truncate()</code> + and <code class="ph codeph">dtrunc()</code> signatures for this function: + </p> +<pre class="pre codeblock"><code>select truncate(3.45); +----------------+ | truncate(3.45) | +----------------+ | 3 | +----------------+ -select truncate(-3.45) +select truncate(-3.45); +-----------------+ | truncate(-3.45) | +-----------------+ | -3 | +-----------------+ -select truncate(3.456,1) +select truncate(3.456,1); +--------------------+ | truncate(3.456, 1) | +--------------------+ | 3.4 | +--------------------+ -select dtrunc(3.456,1) +select dtrunc(3.456,1); +------------------+ | dtrunc(3.456, 1) | +------------------+ | 3.4 | +------------------+ -select truncate(3.456,2) +select truncate(3.456,2); +--------------------+ | truncate(3.456, 2) | +--------------------+ | 3.45 | +--------------------+ -select truncate(3.456,7) +select truncate(3.456,7); +--------------------+ | truncate(3.456, 7) | +--------------------+ | 3.4560000 | +--------------------+ </code></pre> + <p class="p"> + The following examples demonstrate using <code class="ph codeph">trunc()</code> with + <code class="ph codeph">DECIMAL</code> or <code class="ph codeph">DOUBLE</code> values, and with + an optional scale argument for <code class="ph codeph">DECIMAL</code> values. + (The behavior is the same for the <code class="ph codeph">truncate()</code> and + <code class="ph codeph">dtrunc()</code> aliases also.) + </p> +<pre class="pre codeblock"><code> +create table t1 (d decimal(20,7)); + +-- By default, no digits to the right of the decimal point. +insert into t1 values (1.1), (2.22), (3.333), (4.4444), (5.55555); +select trunc(d) from t1 order by d; ++----------+ +| trunc(d) | ++----------+ +| 1 | +| 2 | +| 3 | +| 4 | +| 5 | ++----------+ + +-- 1 digit to the right of the decimal point. +select trunc(d,1) from t1 order by d; ++-------------+ +| trunc(d, 1) | ++-------------+ +| 1.1 | +| 2.2 | +| 3.3 | +| 4.4 | +| 5.5 | ++-------------+ + +-- 2 digits to the right of the decimal point, +-- including trailing zeroes if needed. +select trunc(d,2) from t1 order by d; ++-------------+ +| trunc(d, 2) | ++-------------+ +| 1.10 | +| 2.22 | +| 3.33 | +| 4.44 | +| 5.55 | ++-------------+ + +insert into t1 values (9999.9999), (8888.8888); + +-- Negative scale truncates digits to the left +-- of the decimal point. +select trunc(d,-2) from t1 where d > 100 order by d; ++--------------+ +| trunc(d, -2) | ++--------------+ +| 8800 | +| 9900 | ++--------------+ + +-- The scale of the result is adjusted to match the +-- scale argument. +select trunc(d,2), + precision(trunc(d,2)) as p, + scale(trunc(d,2)) as s +from t1 order by d; ++-------------+----+---+ +| trunc(d, 2) | p | s | ++-------------+----+---+ +| 1.10 | 15 | 2 | +| 2.22 | 15 | 2 | +| 3.33 | 15 | 2 | +| 4.44 | 15 | 2 | +| 5.55 | 15 | 2 | +| 8888.88 | 15 | 2 | +| 9999.99 | 15 | 2 | ++-------------+----+---+ +</code></pre> + +<pre class="pre codeblock"><code> +create table dbl (d double); + +insert into dbl values + (1.1), (2.22), (3.333), (4.4444), (5.55555), + (8888.8888), (9999.9999); + +-- With double values, there is no optional scale argument. +select trunc(d) from dbl order by d; ++----------+ +| trunc(d) | ++----------+ +| 1 | +| 2 | +| 3 | +| 4 | +| 5 | +| 8888 | +| 9999 | ++----------+ +</code></pre> </dd> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ca9005be/docs/build/html/topics/impala_max_row_size.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_max_row_size.html b/docs/build/html/topics/impala_max_row_size.html new file mode 100644 index 0000000..49b3198 --- /dev/null +++ b/docs/build/html/topics/impala_max_row_size.html @@ -0,0 +1,221 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="max_row_size"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>MAX_ROW_SIZE Query Option</title></head><body id="max_row_size"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">MAX_ROW_SIZE Query Option</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Ensures that Impala can process rows of at least the specified size. (Larger + rows might be successfully processed, but that is not guaranteed.) Applies when + constructing intermediate or final rows in the result set. This setting prevents + out-of-control memory use when accessing columns containing huge strings. + </p> + + + + <p class="p"> + <strong class="ph b">Type:</strong> integer + </p> + + <p class="p"> + <strong class="ph b">Default:</strong> + </p> + <p class="p"> + <code class="ph codeph">524288</code> (512 KB) + </p> + + <p class="p"> + <strong class="ph b">Units:</strong> A numeric argument represents a size in bytes; you can also use a suffix of <code class="ph codeph">m</code> + or <code class="ph codeph">mb</code> for megabytes, or <code class="ph codeph">g</code> or <code class="ph codeph">gb</code> for gigabytes. If you + specify a value with unrecognized formats, subsequent queries fail with an error. + </p> + + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.10.0</span> + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + <p class="p"> + If a query fails because it involves rows with long strings and/or + many columns, causing the total row size to exceed <code class="ph codeph">MAX_ROW_SIZE</code> + bytes, increase the <code class="ph codeph">MAX_ROW_SIZE</code> setting to accommodate + the total bytes stored in the largest row. Examine the error messages for any + failed queries to see the size of the row that caused the problem. + </p> + <p class="p"> + Impala attempts to handle rows that exceed the <code class="ph codeph">MAX_ROW_SIZE</code> + value where practical, so in many cases, queries succeed despite having rows + that are larger than this setting. + </p> + <p class="p"> + Specifying a value that is substantially higher than actually needed can cause + Impala to reserve more memory than is necessary to execute the query. + </p> + <p class="p"> + In a Hadoop cluster with highly concurrent workloads and queries that process + high volumes of data, traditional SQL tuning advice about minimizing wasted memory + is worth remembering. For example, if a table has <code class="ph codeph">STRING</code> columns + where a single value might be multiple megabytes, make sure that the + <code class="ph codeph">SELECT</code> lists in queries only refer to columns that are actually + needed in the result set, instead of using the <code class="ph codeph">SELECT *</code> shorthand. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <p class="p"> + The following examples show the kinds of situations where it is necessary to + adjust the <code class="ph codeph">MAX_ROW_SIZE</code> setting. First, we create a table + containing some very long values in <code class="ph codeph">STRING</code> columns: + </p> + +<pre class="pre codeblock"><code> +create table big_strings (s1 string, s2 string, s3 string) stored as parquet; + +-- Turn off compression to more easily reason about data volume by doing SHOW TABLE STATS. +-- Does not actually affect query success or failure, because MAX_ROW_SIZE applies when +-- column values are materialized in memory. +set compression_codec=none; +set; +... + MAX_ROW_SIZE: [524288] +... + +-- A very small row. +insert into big_strings values ('one', 'two', 'three'); +-- A row right around the default MAX_ROW_SIZE limit: a 500 KiB string and a 30 KiB string. +insert into big_strings values (repeat('12345',100000), 'short', repeat('123',10000)); +-- A row that is too big if the query has to materialize both S1 and S3. +insert into big_strings values (repeat('12345',100000), 'short', repeat('12345',100000)); + +</code></pre> + + <p class="p"> + With the default <code class="ph codeph">MAX_ROW_SIZE</code> setting, different queries succeed + or fail based on which column values have to be materialized during query processing: + </p> + +<pre class="pre codeblock"><code> +-- All the S1 values can be materialized within the 512 KB MAX_ROW_SIZE buffer. +select count(distinct s1) from big_strings; ++--------------------+ +| count(distinct s1) | ++--------------------+ +| 2 | ++--------------------+ + +-- A row where even the S1 value is too large to materialize within MAX_ROW_SIZE. +insert into big_strings values (repeat('12345',1000000), 'short', repeat('12345',1000000)); + +-- The 5 MiB string is too large to materialize. The message explains the size of the result +-- set row the query is attempting to materialize. +select count(distinct(s1)) from big_strings; +WARNINGS: Row of size 4.77 MB could not be materialized in plan node with id 1. + Increase the max_row_size query option (currently 512.00 KB) to process larger rows. + +-- If more columns are involved, the result set row being materialized is bigger. +select count(distinct s1, s2, s3) from big_strings; +WARNINGS: Row of size 9.54 MB could not be materialized in plan node with id 1. + Increase the max_row_size query option (currently 512.00 KB) to process larger rows. + +-- Column S2, containing only short strings, can still be examined. +select count(distinct(s2)) from big_strings; ++----------------------+ +| count(distinct (s2)) | ++----------------------+ +| 2 | ++----------------------+ + +-- Queries that do not materialize the big column values are OK. +select count(*) from big_strings; ++----------+ +| count(*) | ++----------+ +| 4 | ++----------+ + +</code></pre> + + <p class="p"> + The following examples show how adjusting <code class="ph codeph">MAX_ROW_SIZE</code> upward + allows queries involving the long string columns to succeed: + </p> + +<pre class="pre codeblock"><code> +-- Boosting MAX_ROW_SIZE moderately allows all S1 values to be materialized. +set max_row_size=7mb; + +select count(distinct s1) from big_strings; ++--------------------+ +| count(distinct s1) | ++--------------------+ +| 3 | ++--------------------+ + +-- But the combination of S1 + S3 strings is still too large. +select count(distinct s1, s2, s3) from big_strings; +WARNINGS: Row of size 9.54 MB could not be materialized in plan node with id 1. Increase the max_row_size query option (currently 7.00 MB) to process larger rows. + +-- Boosting MAX_ROW_SIZE to larger than the largest row in the table allows +-- all queries to complete successfully. +set max_row_size=12mb; + +select count(distinct s1, s2, s3) from big_strings; ++----------------------------+ +| count(distinct s1, s2, s3) | ++----------------------------+ +| 4 | ++----------------------------+ + +</code></pre> + + <p class="p"> + The following examples show how to reason about appropriate values for + <code class="ph codeph">MAX_ROW_SIZE</code>, based on the characteristics of the + columns containing the long values: + </p> + +<pre class="pre codeblock"><code> +-- With a large MAX_ROW_SIZE in place, we can examine the columns to +-- understand the practical lower limit for MAX_ROW_SIZE based on the +-- table structure and column values. +select max(length(s1) + length(s2) + length(s3)) / 1e6 as megabytes from big_strings; ++-----------+ +| megabytes | ++-----------+ +| 10.000005 | ++-----------+ + +-- We can also examine the 'Max Size' for each column after computing stats. +compute stats big_strings; +show column stats big_strings; ++--------+--------+------------------+--------+----------+-----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++--------+--------+------------------+--------+----------+-----------+ +| s1 | STRING | 2 | -1 | 5000000 | 2500002.5 | +| s2 | STRING | 2 | -1 | 10 | 7.5 | +| s3 | STRING | 2 | -1 | 5000000 | 2500005 | ++--------+--------+------------------+--------+----------+-----------+ + +</code></pre> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + <p class="p"> + <a class="xref" href="impala_buffer_pool_limit.html">BUFFER_POOL_LIMIT Query Option</a>, + <a class="xref" href="impala_default_spillable_buffer_size.html">DEFAULT_SPILLABLE_BUFFER_SIZE Query Option</a>, + <a class="xref" href="impala_min_spillable_buffer_size.html">MIN_SPILLABLE_BUFFER_SIZE Query Option</a>, + <a class="xref" href="impala_scalability.html">Scalability Considerations for Impala</a> + </p> + + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav></article></main></body></html> \ No newline at end of file
