http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_components.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_components.html b/docs/build/html/topics/impala_components.html new file mode 100644 index 0000000..d3d210d --- /dev/null +++ b/docs/build/html/topics/impala_components.html @@ -0,0 +1,192 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_concepts.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="intro_components"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Components of the Impala Server</title></head><body id="intro_components"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Components of the Impala Server</h1> + + + + <div class="body conbody"> + + <p class="p"> + The Impala server is a distributed, massively parallel processing (MPP) database engine. It consists of + different daemon processes that run on specific hosts within your <span class="keyword"></span> cluster. + </p> + + <p class="p toc inpage"></p> + </div> + + <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_concepts.html">Impala Concepts and Architecture</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="intro_components__intro_impalad"> + + <h2 class="title topictitle2" id="ariaid-title2">The Impala Daemon</h2> + + <div class="body conbody"> + + <p class="p"> + The core Impala component is a daemon process that runs on each DataNode of the cluster, physically represented + by the <code class="ph codeph">impalad</code> process. It reads and writes to data files; accepts queries transmitted + from the <code class="ph codeph">impala-shell</code> command, Hue, JDBC, or ODBC; parallelizes the queries and + distributes work across the cluster; and transmits intermediate query results back to the + central coordinator node. + </p> + + <p class="p"> + You can submit a query to the Impala daemon running on any DataNode, and that instance of the daemon serves as the + <dfn class="term">coordinator node</dfn> for that query. The other nodes transmit partial results back to the + coordinator, which constructs the final result set for a query. When running experiments with functionality + through the <code class="ph codeph">impala-shell</code> command, you might always connect to the same Impala daemon for + convenience. For clusters running production workloads, you might load-balance by + submitting each query to a different Impala daemon in round-robin style, using the JDBC or ODBC interfaces. + </p> + + <p class="p"> + The Impala daemons are in constant communication with the <dfn class="term">statestore</dfn>, to confirm which nodes + are healthy and can accept new work. + </p> + + <p class="p"> + They also receive broadcast messages from the <span class="keyword cmdname">catalogd</span> daemon (introduced in Impala 1.2) + whenever any Impala node in the cluster creates, alters, or drops any type of object, or when an + <code class="ph codeph">INSERT</code> or <code class="ph codeph">LOAD DATA</code> statement is processed through Impala. This + background communication minimizes the need for <code class="ph codeph">REFRESH</code> or <code class="ph codeph">INVALIDATE + METADATA</code> statements that were needed to coordinate metadata across nodes prior to Impala 1.2. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> <a class="xref" href="impala_config_options.html#config_options">Modifying Impala Startup Options</a>, + <a class="xref" href="impala_processes.html#processes">Starting Impala</a>, <a class="xref" href="impala_timeouts.html#impalad_timeout">Setting the Idle Query and Idle Session Timeouts for impalad</a>, + <a class="xref" href="impala_ports.html#ports">Ports Used by Impala</a>, <a class="xref" href="impala_proxy.html#proxy">Using Impala through a Proxy for High Availability</a> + </p> + </div> + </article> + + <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="intro_components__intro_statestore"> + + <h2 class="title topictitle2" id="ariaid-title3">The Impala Statestore</h2> + + <div class="body conbody"> + + <p class="p"> + The Impala component known as the <dfn class="term">statestore</dfn> checks on the health of Impala daemons on all the + DataNodes in a cluster, and continuously relays its findings to each of those daemons. It is physically + represented by a daemon process named <code class="ph codeph">statestored</code>; you only need such a process on one + host in the cluster. If an Impala daemon goes offline due to hardware failure, network error, software issue, + or other reason, the statestore informs all the other Impala daemons so that future queries can avoid making + requests to the unreachable node. + </p> + + <p class="p"> + Because the statestore's purpose is to help when things go wrong, it is not critical to the normal + operation of an Impala cluster. If the statestore is not running or becomes unreachable, the Impala daemons + continue running and distributing work among themselves as usual; the cluster just becomes less robust if + other Impala daemons fail while the statestore is offline. When the statestore comes back online, it re-establishes + communication with the Impala daemons and resumes its monitoring function. + </p> + + <p class="p"> + Most considerations for load balancing and high availability apply to the <span class="keyword cmdname">impalad</span> daemon. + The <span class="keyword cmdname">statestored</span> and <span class="keyword cmdname">catalogd</span> daemons do not have special + requirements for high availability, because problems with those daemons do not result in data loss. + If those daemons become unavailable due to an outage on a particular + host, you can stop the Impala service, delete the <span class="ph uicontrol">Impala StateStore</span> and + <span class="ph uicontrol">Impala Catalog Server</span> roles, add the roles on a different host, and restart the + Impala service. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_scalability.html#statestore_scalability">Scalability Considerations for the Impala Statestore</a>, + <a class="xref" href="impala_config_options.html#config_options">Modifying Impala Startup Options</a>, <a class="xref" href="impala_processes.html#processes">Starting Impala</a>, + <a class="xref" href="impala_timeouts.html#statestore_timeout">Increasing the Statestore Timeout</a>, <a class="xref" href="impala_ports.html#ports">Ports Used by Impala</a> + </p> + </div> + </article> + + <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="intro_components__intro_catalogd"> + + <h2 class="title topictitle2" id="ariaid-title4">The Impala Catalog Service</h2> + + <div class="body conbody"> + + <p class="p"> + The Impala component known as the <dfn class="term">catalog service</dfn> relays the metadata changes from Impala SQL + statements to all the DataNodes in a cluster. It is physically represented by a daemon process named + <code class="ph codeph">catalogd</code>; you only need such a process on one host in the cluster. Because the requests + are passed through the statestore daemon, it makes sense to run the <span class="keyword cmdname">statestored</span> and + <span class="keyword cmdname">catalogd</span> services on the same host. + </p> + + <p class="p"> + The catalog service avoids the need to issue + <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> statements when the metadata changes are + performed by statements issued through Impala. When you create a table, load data, and so on through Hive, + you do need to issue <code class="ph codeph">REFRESH</code> or <code class="ph codeph">INVALIDATE METADATA</code> on an Impala node + before executing a query there. + </p> + + <p class="p"> + This feature touches a number of aspects of Impala: + </p> + + + + <ul class="ul" id="intro_catalogd__catalogd_xrefs"> + <li class="li"> + <p class="p"> + See <a class="xref" href="impala_install.html#install">Installing Impala</a>, <a class="xref" href="impala_upgrading.html#upgrading">Upgrading Impala</a> and + <a class="xref" href="impala_processes.html#processes">Starting Impala</a>, for usage information for the + <span class="keyword cmdname">catalogd</span> daemon. + </p> + </li> + + <li class="li"> + <p class="p"> + The <code class="ph codeph">REFRESH</code> and <code class="ph codeph">INVALIDATE METADATA</code> statements are not needed + when the <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">INSERT</code>, or other table-changing or + data-changing operation is performed through Impala. These statements are still needed if such + operations are done through Hive or by manipulating data files directly in HDFS, but in those cases the + statements only need to be issued on one Impala node rather than on all nodes. See + <a class="xref" href="impala_refresh.html#refresh">REFRESH Statement</a> and + <a class="xref" href="impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a> for the latest usage information for + those statements. + </p> + </li> + </ul> + + <p class="p"> + By default, the metadata loading and caching on startup happens asynchronously, so Impala can begin + accepting requests promptly. To enable the original behavior, where Impala waited until all metadata was + loaded before accepting any requests, set the <span class="keyword cmdname">catalogd</span> configuration option + <code class="ph codeph">--load_catalog_in_background=false</code>. + </p> + + <p class="p"> + Most considerations for load balancing and high availability apply to the <span class="keyword cmdname">impalad</span> daemon. + The <span class="keyword cmdname">statestored</span> and <span class="keyword cmdname">catalogd</span> daemons do not have special + requirements for high availability, because problems with those daemons do not result in data loss. + If those daemons become unavailable due to an outage on a particular + host, you can stop the Impala service, delete the <span class="ph uicontrol">Impala StateStore</span> and + <span class="ph uicontrol">Impala Catalog Server</span> roles, add the roles on a different host, and restart the + Impala service. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + In Impala 1.2.4 and higher, you can specify a table name with <code class="ph codeph">INVALIDATE METADATA</code> after + the table is created in Hive, allowing you to make individual tables visible to Impala without doing a full + reload of the catalog metadata. Impala 1.2.4 also includes other changes to make the metadata broadcast + mechanism faster and more responsive, especially during Impala startup. See + <a class="xref" href="../shared/../topics/impala_new_features.html#new_features_124">New Features in Impala 1.2.4</a> for details. + </p> + </div> + + <p class="p"> + <strong class="ph b">Related information:</strong> <a class="xref" href="impala_config_options.html#config_options">Modifying Impala Startup Options</a>, + <a class="xref" href="impala_processes.html#processes">Starting Impala</a>, <a class="xref" href="impala_ports.html#ports">Ports Used by Impala</a> + </p> + </div> + </article> +</article></main></body></html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_compression_codec.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_compression_codec.html b/docs/build/html/topics/impala_compression_codec.html new file mode 100644 index 0000000..2018299 --- /dev/null +++ b/docs/build/html/topics/impala_compression_codec.html @@ -0,0 +1,92 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="compression_codec"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>COMPRESSION_CODEC Query Option (Impala 2.0 or higher only)</title></head><body id="compression_codec"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">COMPRESSION_CODEC Query Option (<span class="keyword">Impala 2.0</span> or higher only)</h1> + + + + <div class="body conbody"> + + + + + + <p class="p"> + + When Impala writes Parquet data files using the <code class="ph codeph">INSERT</code> statement, the underlying compression + is controlled by the <code class="ph codeph">COMPRESSION_CODEC</code> query option. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + Prior to Impala 2.0, this option was named <code class="ph codeph">PARQUET_COMPRESSION_CODEC</code>. In Impala 2.0 and + later, the <code class="ph codeph">PARQUET_COMPRESSION_CODEC</code> name is not recognized. Use the more general name + <code class="ph codeph">COMPRESSION_CODEC</code> for new code. + </div> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>SET COMPRESSION_CODEC=<var class="keyword varname">codec_name</var>;</code></pre> + + <p class="p"> + The allowed values for this query option are <code class="ph codeph">SNAPPY</code> (the default), <code class="ph codeph">GZIP</code>, + and <code class="ph codeph">NONE</code>. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + A Parquet file created with <code class="ph codeph">COMPRESSION_CODEC=NONE</code> is still typically smaller than the + original data, due to encoding schemes such as run-length encoding and dictionary encoding that are applied + separately from compression. + </div> + + <p class="p"></p> + + <p class="p"> + The option value is not case-sensitive. + </p> + + <p class="p"> + If the option is set to an unrecognized value, all kinds of queries will fail due to the invalid option + setting, not just queries involving Parquet tables. (The value <code class="ph codeph">BZIP2</code> is also recognized, but + is not compatible with Parquet tables.) + </p> + + <p class="p"> + <strong class="ph b">Type:</strong> <code class="ph codeph">STRING</code> + </p> + + <p class="p"> + <strong class="ph b">Default:</strong> <code class="ph codeph">SNAPPY</code> + </p> + + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + +<pre class="pre codeblock"><code>set compression_codec=gzip; +insert into parquet_table_highly_compressed select * from t1; + +set compression_codec=snappy; +insert into parquet_table_compression_plus_fast_queries select * from t1; + +set compression_codec=none; +insert into parquet_table_no_compression select * from t1; + +set compression_codec=foo; +select * from t1 limit 5; +ERROR: Invalid compression codec: foo +</code></pre> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + For information about how compressing Parquet data files affects query performance, see + <a class="xref" href="impala_parquet.html#parquet_compression">Snappy and GZip Compression for Parquet Data Files</a>. + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_compute_stats.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_compute_stats.html b/docs/build/html/topics/impala_compute_stats.html new file mode 100644 index 0000000..fcba3d6 --- /dev/null +++ b/docs/build/html/topics/impala_compute_stats.html @@ -0,0 +1,558 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="compute_stats"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>COMPUTE STATS Statement</title></head><body id="compute_stats"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">COMPUTE STATS Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Gathers information about volume and distribution of data in a table and all associated columns and + partitions. The information is stored in the metastore database, and used by Impala to help optimize queries. + For example, if Impala can determine that a table is large or small, or has many or few distinct values it + can organize parallelize the work appropriately for a join query or insert operation. For details about the + kinds of information gathered by this statement, see <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a>. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>COMPUTE STATS [<var class="keyword varname">db_name</var>.]<var class="keyword varname">table_name</var> +COMPUTE INCREMENTAL STATS [<var class="keyword varname">db_name</var>.]<var class="keyword varname">table_name</var> [PARTITION (<var class="keyword varname">partition_spec</var>)] + +<var class="keyword varname">partition_spec</var> ::= <var class="keyword varname">simple_partition_spec</var> | <span class="ph"><var class="keyword varname">complex_partition_spec</var></span> + +<var class="keyword varname">simple_partition_spec</var> ::= <var class="keyword varname">partition_col</var>=<var class="keyword varname">constant_value</var> + +<span class="ph"><var class="keyword varname">complex_partition_spec</var> ::= <var class="keyword varname">comparison_expression_on_partition_col</var></span> +</code></pre> + + <p class="p"> + The <code class="ph codeph">PARTITION</code> clause is only allowed in combination with the <code class="ph codeph">INCREMENTAL</code> + clause. It is optional for <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>, and required for <code class="ph codeph">DROP + INCREMENTAL STATS</code>. Whenever you specify partitions through the <code class="ph codeph">PARTITION + (<var class="keyword varname">partition_spec</var>)</code> clause in a <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> or + <code class="ph codeph">DROP INCREMENTAL STATS</code> statement, you must include all the partitioning columns in the + specification, and specify constant values for all the partition key columns. + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + Originally, Impala relied on users to run the Hive <code class="ph codeph">ANALYZE TABLE</code> statement, but that method + of gathering statistics proved unreliable and difficult to use. The Impala <code class="ph codeph">COMPUTE STATS</code> + statement is built from the ground up to improve the reliability and user-friendliness of this operation. + <code class="ph codeph">COMPUTE STATS</code> does not require any setup steps or special configuration. You only run a + single Impala <code class="ph codeph">COMPUTE STATS</code> statement to gather both table and column statistics, rather + than separate Hive <code class="ph codeph">ANALYZE TABLE</code> statements for each kind of statistics. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> variation is a shortcut for partitioned tables that works on a + subset of partitions rather than the entire table. The incremental nature makes it suitable for large tables + with many partitions, where a full <code class="ph codeph">COMPUTE STATS</code> operation takes too long to be practical + each time a partition is added or dropped. See <a class="xref" href="impala_perf_stats.html#perf_stats_incremental">Overview of Incremental Statistics</a> + for full usage details. + </p> + + <p class="p"> + <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> only applies to partitioned tables. If you use the + <code class="ph codeph">INCREMENTAL</code> clause for an unpartitioned table, Impala automatically uses the original + <code class="ph codeph">COMPUTE STATS</code> statement. Such tables display <code class="ph codeph">false</code> under the + <code class="ph codeph">Incremental stats</code> column of the <code class="ph codeph">SHOW TABLE STATS</code> output. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + Because many of the most performance-critical and resource-intensive operations rely on table and column + statistics to construct accurate and efficient plans, <code class="ph codeph">COMPUTE STATS</code> is an important step at + the end of your ETL process. Run <code class="ph codeph">COMPUTE STATS</code> on all tables as your first step during + performance tuning for slow queries, or troubleshooting for out-of-memory conditions: + <ul class="ul"> + <li class="li"> + Accurate statistics help Impala construct an efficient query plan for join queries, improving performance + and reducing memory usage. + </li> + + <li class="li"> + Accurate statistics help Impala distribute the work effectively for insert operations into Parquet + tables, improving performance and reducing memory usage. + </li> + + <li class="li"> + Accurate statistics help Impala estimate the memory required for each query, which is important when you + use resource management features, such as admission control and the YARN resource management framework. + The statistics help Impala to achieve high concurrency, full utilization of available memory, and avoid + contention with workloads from other Hadoop components. + </li> + <li class="li"> + In <span class="keyword">Impala 2.8</span> and higher, when you run the + <code class="ph codeph">COMPUTE STATS</code> or <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> + statement against a Parquet table, Impala automatically applies the query + option setting <code class="ph codeph">MT_DOP=4</code> to increase the amount of intra-node + parallelism during this CPU-intensive operation. See <a class="xref" href="impala_mt_dop.html">MT_DOP Query Option</a> + for details about what this query option does and how to use it with + CPU-intensive <code class="ph codeph">SELECT</code> statements. + </li> + </ul> + </div> + + <p class="p"> + <strong class="ph b">Computing stats for groups of partitions:</strong> + </p> + + <p class="p"> + In <span class="keyword">Impala 2.8</span> and higher, you can run <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> + on multiple partitions, instead of the entire table or one partition at a time. You include + comparison operators other than <code class="ph codeph">=</code> in the <code class="ph codeph">PARTITION</code> clause, + and the <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement applies to all partitions that + match the comparison expression. + </p> + + <p class="p"> + For example, the <code class="ph codeph">INT_PARTITIONS</code> table contains 4 partitions. + The following <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statements affect some but not all + partitions, as indicated by the <code class="ph codeph">Updated <var class="keyword varname">n</var> partition(s)</code> + messages. The partitions that are affected depend on values in the partition key column <code class="ph codeph">X</code> + that match the comparison expression in the <code class="ph codeph">PARTITION</code> clause. + </p> + +<pre class="pre codeblock"><code> +show partitions int_partitions; ++-------+-------+--------+------+--------------+-------------------+---------+... +| x | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |... ++-------+-------+--------+------+--------------+-------------------+---------+... +| 99 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | PARQUET |... +| 120 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT |... +| 150 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT |... +| 200 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT |... +| Total | -1 | 0 | 0B | 0B | | |... ++-------+-------+--------+------+--------------+-------------------+---------+... + +compute incremental stats int_partitions partition (x < 100); ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 1 partition(s) and 1 column(s). | ++-----------------------------------------+ + +compute incremental stats int_partitions partition (x in (100, 150, 200)); ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 2 partition(s) and 1 column(s). | ++-----------------------------------------+ + +compute incremental stats int_partitions partition (x between 100 and 175); ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 2 partition(s) and 1 column(s). | ++-----------------------------------------+ + +compute incremental stats int_partitions partition (x in (100, 150, 200) or x < 100); ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 3 partition(s) and 1 column(s). | ++-----------------------------------------+ + +compute incremental stats int_partitions partition (x != 150); ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 3 partition(s) and 1 column(s). | ++-----------------------------------------+ + +</code></pre> + + <p class="p"> + <strong class="ph b">Complex type considerations:</strong> + </p> + + <p class="p"> + Currently, the statistics created by the <code class="ph codeph">COMPUTE STATS</code> statement do not include + information about complex type columns. The column stats metrics for complex columns are always shown + as -1. For queries involving complex type columns, Impala uses + heuristics to estimate the data distribution within such columns. + </p> + + <p class="p"> + <strong class="ph b">HBase considerations:</strong> + </p> + + <p class="p"> + <code class="ph codeph">COMPUTE STATS</code> works for HBase tables also. The statistics gathered for HBase tables are + somewhat different than for HDFS-backed tables, but that metadata is still used for optimization when HBase + tables are involved in join queries. + </p> + + <p class="p"> + <strong class="ph b">Amazon S3 considerations:</strong> + </p> + + <p class="p"> + <code class="ph codeph">COMPUTE STATS</code> also works for tables where data resides in the Amazon Simple Storage Service (S3). + See <a class="xref" href="impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details. + </p> + + <p class="p"> + <strong class="ph b">Performance considerations:</strong> + </p> + + <p class="p"> + The statistics collected by <code class="ph codeph">COMPUTE STATS</code> are used to optimize join queries + <code class="ph codeph">INSERT</code> operations into Parquet tables, and other resource-intensive kinds of SQL statements. + See <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a> for details. + </p> + + <p class="p"> + For large tables, the <code class="ph codeph">COMPUTE STATS</code> statement itself might take a long time and you + might need to tune its performance. The <code class="ph codeph">COMPUTE STATS</code> statement does not work with the + <code class="ph codeph">EXPLAIN</code> statement, or the <code class="ph codeph">SUMMARY</code> command in <span class="keyword cmdname">impala-shell</span>. + You can use the <code class="ph codeph">PROFILE</code> statement in <span class="keyword cmdname">impala-shell</span> to examine timing information + for the statement as a whole. If a basic <code class="ph codeph">COMPUTE STATS</code> statement takes a long time for a + partitioned table, consider switching to the <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> syntax so that only + newly added partitions are analyzed each time. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <p class="p"> + This example shows two tables, <code class="ph codeph">T1</code> and <code class="ph codeph">T2</code>, with a small number distinct + values linked by a parent-child relationship between <code class="ph codeph">T1.ID</code> and <code class="ph codeph">T2.PARENT</code>. + <code class="ph codeph">T1</code> is tiny, while <code class="ph codeph">T2</code> has approximately 100K rows. Initially, the statistics + includes physical measurements such as the number of files, the total size, and size measurements for + fixed-length columns such as with the <code class="ph codeph">INT</code> type. Unknown values are represented by -1. After + running <code class="ph codeph">COMPUTE STATS</code> for each table, much more information is available through the + <code class="ph codeph">SHOW STATS</code> statements. If you were running a join query involving both of these tables, you + would need statistics for both tables to get the most effective optimization for the query. + </p> + + + +<pre class="pre codeblock"><code>[localhost:21000] > show table stats t1; +Query: show table stats t1 ++-------+--------+------+--------+ +| #Rows | #Files | Size | Format | ++-------+--------+------+--------+ +| -1 | 1 | 33B | TEXT | ++-------+--------+------+--------+ +Returned 1 row(s) in 0.02s +[localhost:21000] > show table stats t2; +Query: show table stats t2 ++-------+--------+----------+--------+ +| #Rows | #Files | Size | Format | ++-------+--------+----------+--------+ +| -1 | 28 | 960.00KB | TEXT | ++-------+--------+----------+--------+ +Returned 1 row(s) in 0.01s +[localhost:21000] > show column stats t1; +Query: show column stats t1 ++--------+--------+------------------+--------+----------+----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++--------+--------+------------------+--------+----------+----------+ +| id | INT | -1 | -1 | 4 | 4 | +| s | STRING | -1 | -1 | -1 | -1 | ++--------+--------+------------------+--------+----------+----------+ +Returned 2 row(s) in 1.71s +[localhost:21000] > show column stats t2; +Query: show column stats t2 ++--------+--------+------------------+--------+----------+----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++--------+--------+------------------+--------+----------+----------+ +| parent | INT | -1 | -1 | 4 | 4 | +| s | STRING | -1 | -1 | -1 | -1 | ++--------+--------+------------------+--------+----------+----------+ +Returned 2 row(s) in 0.01s +[localhost:21000] > compute stats t1; +Query: compute stats t1 ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 1 partition(s) and 2 column(s). | ++-----------------------------------------+ +Returned 1 row(s) in 5.30s +[localhost:21000] > show table stats t1; +Query: show table stats t1 ++-------+--------+------+--------+ +| #Rows | #Files | Size | Format | ++-------+--------+------+--------+ +| 3 | 1 | 33B | TEXT | ++-------+--------+------+--------+ +Returned 1 row(s) in 0.01s +[localhost:21000] > show column stats t1; +Query: show column stats t1 ++--------+--------+------------------+--------+----------+----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++--------+--------+------------------+--------+----------+----------+ +| id | INT | 3 | -1 | 4 | 4 | +| s | STRING | 3 | -1 | -1 | -1 | ++--------+--------+------------------+--------+----------+----------+ +Returned 2 row(s) in 0.02s +[localhost:21000] > compute stats t2; +Query: compute stats t2 ++-----------------------------------------+ +| summary | ++-----------------------------------------+ +| Updated 1 partition(s) and 2 column(s). | ++-----------------------------------------+ +Returned 1 row(s) in 5.70s +[localhost:21000] > show table stats t2; +Query: show table stats t2 ++-------+--------+----------+--------+ +| #Rows | #Files | Size | Format | ++-------+--------+----------+--------+ +| 98304 | 1 | 960.00KB | TEXT | ++-------+--------+----------+--------+ +Returned 1 row(s) in 0.03s +[localhost:21000] > show column stats t2; +Query: show column stats t2 ++--------+--------+------------------+--------+----------+----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++--------+--------+------------------+--------+----------+----------+ +| parent | INT | 3 | -1 | 4 | 4 | +| s | STRING | 6 | -1 | 14 | 9.3 | ++--------+--------+------------------+--------+----------+----------+ +Returned 2 row(s) in 0.01s</code></pre> + + <p class="p"> + The following example shows how to use the <code class="ph codeph">INCREMENTAL</code> clause, available in Impala 2.1.0 and + higher. The <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> syntax lets you collect statistics for newly added or + changed partitions, without rescanning the entire table. + </p> + +<pre class="pre codeblock"><code>-- Initially the table has no incremental stats, as indicated +-- by -1 under #Rows and false under Incremental stats. +show table stats item_partitioned; ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+------------------ +| Books | -1 | 1 | 223.74KB | NOT CACHED | PARQUET | false +| Children | -1 | 1 | 230.05KB | NOT CACHED | PARQUET | false +| Electronics | -1 | 1 | 232.67KB | NOT CACHED | PARQUET | false +| Home | -1 | 1 | 232.56KB | NOT CACHED | PARQUET | false +| Jewelry | -1 | 1 | 223.72KB | NOT CACHED | PARQUET | false +| Men | -1 | 1 | 231.25KB | NOT CACHED | PARQUET | false +| Music | -1 | 1 | 237.90KB | NOT CACHED | PARQUET | false +| Shoes | -1 | 1 | 234.90KB | NOT CACHED | PARQUET | false +| Sports | -1 | 1 | 227.97KB | NOT CACHED | PARQUET | false +| Women | -1 | 1 | 226.27KB | NOT CACHED | PARQUET | false +| Total | -1 | 10 | 2.25MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+------------------ + +-- After the first COMPUTE INCREMENTAL STATS, +-- all partitions have stats. +compute incremental stats item_partitioned; ++-------------------------------------------+ +| summary | ++-------------------------------------------+ +| Updated 10 partition(s) and 21 column(s). | ++-------------------------------------------+ +show table stats item_partitioned; ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+------------------ +| Books | 1733 | 1 | 223.74KB | NOT CACHED | PARQUET | true +| Children | 1786 | 1 | 230.05KB | NOT CACHED | PARQUET | true +| Electronics | 1812 | 1 | 232.67KB | NOT CACHED | PARQUET | true +| Home | 1807 | 1 | 232.56KB | NOT CACHED | PARQUET | true +| Jewelry | 1740 | 1 | 223.72KB | NOT CACHED | PARQUET | true +| Men | 1811 | 1 | 231.25KB | NOT CACHED | PARQUET | true +| Music | 1860 | 1 | 237.90KB | NOT CACHED | PARQUET | true +| Shoes | 1835 | 1 | 234.90KB | NOT CACHED | PARQUET | true +| Sports | 1783 | 1 | 227.97KB | NOT CACHED | PARQUET | true +| Women | 1790 | 1 | 226.27KB | NOT CACHED | PARQUET | true +| Total | 17957 | 10 | 2.25MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+------------------ + +-- Add a new partition... +alter table item_partitioned add partition (i_category='Camping'); +-- Add or replace files in HDFS outside of Impala, +-- rendering the stats for a partition obsolete. +!import_data_into_sports_partition.sh +refresh item_partitioned; +drop incremental stats item_partitioned partition (i_category='Sports'); +-- Now some partitions have incremental stats +-- and some do not. +show table stats item_partitioned; ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+------------------ +| Books | 1733 | 1 | 223.74KB | NOT CACHED | PARQUET | true +| Camping | -1 | 1 | 408.02KB | NOT CACHED | PARQUET | false +| Children | 1786 | 1 | 230.05KB | NOT CACHED | PARQUET | true +| Electronics | 1812 | 1 | 232.67KB | NOT CACHED | PARQUET | true +| Home | 1807 | 1 | 232.56KB | NOT CACHED | PARQUET | true +| Jewelry | 1740 | 1 | 223.72KB | NOT CACHED | PARQUET | true +| Men | 1811 | 1 | 231.25KB | NOT CACHED | PARQUET | true +| Music | 1860 | 1 | 237.90KB | NOT CACHED | PARQUET | true +| Shoes | 1835 | 1 | 234.90KB | NOT CACHED | PARQUET | true +| Sports | -1 | 1 | 227.97KB | NOT CACHED | PARQUET | false +| Women | 1790 | 1 | 226.27KB | NOT CACHED | PARQUET | true +| Total | 17957 | 11 | 2.65MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+------------------ + +-- After another COMPUTE INCREMENTAL STATS, +-- all partitions have incremental stats, and only the 2 +-- partitions without incremental stats were scanned. +compute incremental stats item_partitioned; ++------------------------------------------+ +| summary | ++------------------------------------------+ +| Updated 2 partition(s) and 21 column(s). | ++------------------------------------------+ +show table stats item_partitioned; ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+------------------ +| Books | 1733 | 1 | 223.74KB | NOT CACHED | PARQUET | true +| Camping | 5328 | 1 | 408.02KB | NOT CACHED | PARQUET | true +| Children | 1786 | 1 | 230.05KB | NOT CACHED | PARQUET | true +| Electronics | 1812 | 1 | 232.67KB | NOT CACHED | PARQUET | true +| Home | 1807 | 1 | 232.56KB | NOT CACHED | PARQUET | true +| Jewelry | 1740 | 1 | 223.72KB | NOT CACHED | PARQUET | true +| Men | 1811 | 1 | 231.25KB | NOT CACHED | PARQUET | true +| Music | 1860 | 1 | 237.90KB | NOT CACHED | PARQUET | true +| Shoes | 1835 | 1 | 234.90KB | NOT CACHED | PARQUET | true +| Sports | 1783 | 1 | 227.97KB | NOT CACHED | PARQUET | true +| Women | 1790 | 1 | 226.27KB | NOT CACHED | PARQUET | true +| Total | 17957 | 11 | 2.65MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+------------------ +</code></pre> + + <p class="p"> + <strong class="ph b">File format considerations:</strong> + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with tables created with any of the file formats supported + by Impala. See <a class="xref" href="impala_file_formats.html#file_formats">How Impala Works with Hadoop File Formats</a> for details about working with the + different file formats. The following considerations apply to <code class="ph codeph">COMPUTE STATS</code> depending on the + file format of the table. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with text tables with no restrictions. These tables can be + created through either Impala or Hive. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with Parquet tables. These tables can be created through + either Impala or Hive. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with Avro tables without restriction in <span class="keyword">Impala 2.2</span> + and higher. In earlier releases, <code class="ph codeph">COMPUTE STATS</code> worked only for Avro tables created through Hive, + and required the <code class="ph codeph">CREATE TABLE</code> statement to use SQL-style column names and types rather than an + Avro-style schema specification. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with RCFile tables with no restrictions. These tables can + be created through either Impala or Hive. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with SequenceFile tables with no restrictions. These + tables can be created through either Impala or Hive. + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement works with partitioned tables, whether all the partitions use + the same file format, or some partitions are defined through <code class="ph codeph">ALTER TABLE</code> to use different + file formats. + </p> + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Certain multi-stage statements (<code class="ph codeph">CREATE TABLE AS SELECT</code> and + <code class="ph codeph">COMPUTE STATS</code>) can be cancelled during some stages, when running <code class="ph codeph">INSERT</code> + or <code class="ph codeph">SELECT</code> operations internally. To cancel this statement, use Ctrl-C from the + <span class="keyword cmdname">impala-shell</span> interpreter, the <span class="ph uicontrol">Cancel</span> button from the + <span class="ph uicontrol">Watch</span> page in Hue, or <span class="ph uicontrol">Cancel</span> from the list of + in-flight queries (for a particular node) on the <span class="ph uicontrol">Queries</span> tab in the Impala web UI + (port 25000). + </p> + + <p class="p"> + <strong class="ph b">Restrictions:</strong> + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> Prior to Impala 1.4.0, + <code class="ph codeph">COMPUTE STATS</code> counted the number of + <code class="ph codeph">NULL</code> values in each column and recorded that figure + in the metastore database. Because Impala does not currently use the + <code class="ph codeph">NULL</code> count during query planning, Impala 1.4.0 and + higher speeds up the <code class="ph codeph">COMPUTE STATS</code> statement by + skipping this <code class="ph codeph">NULL</code> counting. </div> + + <p class="p"> + <strong class="ph b">Internal details:</strong> + </p> + <p class="p"> + Behind the scenes, the <code class="ph codeph">COMPUTE STATS</code> statement + executes two statements: one to count the rows of each partition + in the table (or the entire table if unpartitioned) through the + <code class="ph codeph">COUNT(*)</code> function, + and another to count the approximate number of distinct values + in each column through the <code class="ph codeph">NDV()</code> function. + You might see these queries in your monitoring and diagnostic displays. + The same factors that affect the performance, scalability, and + execution of other queries (such as parallel execution, memory usage, + admission control, and timeouts) also apply to the queries run by the + <code class="ph codeph">COMPUTE STATS</code> statement. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, must have read + permission for all affected files in the source directory: + all files in the case of an unpartitioned table or + a partitioned table in the case of <code class="ph codeph">COMPUTE STATS</code>; + or all the files in partitions without incremental stats in + the case of <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>. + It must also have read and execute permissions for all + relevant directories holding the data files. + (Essentially, <code class="ph codeph">COMPUTE STATS</code> requires the + same permissions as the underlying <code class="ph codeph">SELECT</code> queries it runs + against the table.) + </p> + + <p class="p"> + <strong class="ph b">Kudu considerations:</strong> + </p> + + <p class="p"> + The <code class="ph codeph">COMPUTE STATS</code> statement applies to Kudu tables. + Impala does not compute the number of rows for each partition for + Kudu tables. Therefore, you do not need to re-run the operation when + you see -1 in the <code class="ph codeph"># Rows</code> column of the output from + <code class="ph codeph">SHOW TABLE STATS</code>. That column always shows -1 for + all Kudu tables. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_drop_stats.html#drop_stats">DROP STATS Statement</a>, <a class="xref" href="impala_show.html#show_table_stats">SHOW TABLE STATS Statement</a>, + <a class="xref" href="impala_show.html#show_column_stats">SHOW COLUMN STATS Statement</a>, <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_concepts.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_concepts.html b/docs/build/html/topics/impala_concepts.html new file mode 100644 index 0000000..e644e93 --- /dev/null +++ b/docs/build/html/topics/impala_concepts.html @@ -0,0 +1,48 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_components.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_development.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_hadoop.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="concepts"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala Concepts and Architecture</title></head><body id="concepts"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Impala Concepts and Architecture</h1> + + + + <div class="body conbody"> + + <p class="p"> + The following sections provide background information to help you become productive using Impala and + its features. Where appropriate, the explanations include context to help understand how aspects of Impala + relate to other technologies you might already be familiar with, such as relational database management + systems and data warehouses, or other Hadoop components such as Hive, HDFS, and HBase. + </p> + + <p class="p toc"></p> + </div> + + + + + + + + + + + + + + + + + + + + + + + + + + + + +<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_components.html">Components of the Impala Server</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_development.html">Developing Impala Applications</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_hadoop.html">How Impala Fits Into the Hadoop Ecosystem</a></strong><br></li></ul></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_conditional_functions.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_conditional_functions.html b/docs/build/html/topics/impala_conditional_functions.html new file mode 100644 index 0000000..713946b --- /dev/null +++ b/docs/build/html/topics/impala_conditional_functions.html @@ -0,0 +1,517 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_functions.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="conditional_functions"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Impala Conditional Functions</title></head><body id="conditional_functions"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Impala Conditional Functions</h1> + + + + <div class="body conbody"> + + <p class="p"> + Impala supports the following conditional functions for testing equality, comparison operators, and nullity: + </p> + + <dl class="dl"> + + + <dt class="dt dlterm" id="conditional_functions__case"> + <code class="ph codeph">CASE a WHEN b THEN c [WHEN d THEN e]... [ELSE f] END</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Compares an expression to one or more possible values, and returns a corresponding result + when a match is found. + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + <p class="p"> + In this form of the <code class="ph codeph">CASE</code> expression, the initial value <code class="ph codeph">A</code> + being evaluated for each row it typically a column reference, or an expression involving + a column. This form can only compare against a set of specified values, not ranges, + multi-value comparisons such as <code class="ph codeph">BETWEEN</code> or <code class="ph codeph">IN</code>, + regular expressions, or <code class="ph codeph">NULL</code>. + </p> + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + <p class="p"> + Although this example is split across multiple lines, you can put any or all parts of a <code class="ph codeph">CASE</code> expression + on a single line, with no punctuation or other separators between the <code class="ph codeph">WHEN</code>, + <code class="ph codeph">ELSE</code>, and <code class="ph codeph">END</code> clauses. + </p> +<pre class="pre codeblock"><code>select case x + when 1 then 'one' + when 2 then 'two' + when 0 then 'zero' + else 'out of range' + end + from t1; +</code></pre> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__case2"> + <code class="ph codeph">CASE WHEN a THEN b [WHEN c THEN d]... [ELSE e] END</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests whether any of a sequence of expressions is true, and returns a corresponding + result for the first true expression. + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + <p class="p"> + <code class="ph codeph">CASE</code> expressions without an initial test value have more flexibility. + For example, they can test different columns in different <code class="ph codeph">WHEN</code> clauses, + or use comparison operators such as <code class="ph codeph">BETWEEN</code>, <code class="ph codeph">IN</code> and <code class="ph codeph">IS NULL</code> + rather than comparing against discrete values. + </p> + <p class="p"> + <code class="ph codeph">CASE</code> expressions are often the foundation of long queries that + summarize and format results for easy-to-read reports. For example, you might + use a <code class="ph codeph">CASE</code> function call to turn values from a numeric column + into category strings corresponding to integer values, or labels such as <span class="q">"Small"</span>, + <span class="q">"Medium"</span> and <span class="q">"Large"</span> based on ranges. Then subsequent parts of the + query might aggregate based on the transformed values, such as how many + values are classified as small, medium, or large. You can also use <code class="ph codeph">CASE</code> + to signal problems with out-of-bounds values, <code class="ph codeph">NULL</code> values, + and so on. + </p> + <p class="p"> + By using operators such as <code class="ph codeph">OR</code>, <code class="ph codeph">IN</code>, + <code class="ph codeph">REGEXP</code>, and so on in <code class="ph codeph">CASE</code> expressions, + you can build extensive tests and transformations into a single query. + Therefore, applications that construct SQL statements often rely heavily on <code class="ph codeph">CASE</code> + calls in the generated SQL code. + </p> + <p class="p"> + Because this flexible form of the <code class="ph codeph">CASE</code> expressions allows you to perform + many comparisons and call multiple functions when evaluating each row, be careful applying + elaborate <code class="ph codeph">CASE</code> expressions to queries that process large amounts of data. + For example, when practical, evaluate and transform values through <code class="ph codeph">CASE</code> + after applying operations such as aggregations that reduce the size of the result set; + transform numbers to strings after performing joins with the original numeric values. + </p> + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + <p class="p"> + Although this example is split across multiple lines, you can put any or all parts of a <code class="ph codeph">CASE</code> expression + on a single line, with no punctuation or other separators between the <code class="ph codeph">WHEN</code>, + <code class="ph codeph">ELSE</code>, and <code class="ph codeph">END</code> clauses. + </p> +<pre class="pre codeblock"><code>select case + when dayname(now()) in ('Saturday','Sunday') then 'result undefined on weekends' + when x > y then 'x greater than y' + when x = y then 'x and y are equal' + when x is null or y is null then 'one of the columns is null' + else null + end + from t1; +</code></pre> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__coalesce"> + <code class="ph codeph">coalesce(type v1, type v2, ...)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Returns the first specified argument that is not <code class="ph codeph">NULL</code>, or + <code class="ph codeph">NULL</code> if all arguments are <code class="ph codeph">NULL</code>. + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__decode"> + <code class="ph codeph">decode(type expression, type search1, type result1 [, type search2, type result2 ...] [, type + default] )</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Compares an expression to one or more possible values, and returns a corresponding result + when a match is found. + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + <p class="p"> + Can be used as shorthand for a <code class="ph codeph">CASE</code> expression. + </p> + <p class="p"> + The original expression and the search expressions must of the same type or convertible types. The + result expression can be a different type, but all result expressions must be of the same type. + </p> + <p class="p"> + Returns a successful match If the original expression is <code class="ph codeph">NULL</code> and a search expression + is also <code class="ph codeph">NULL</code>. the + </p> + <p class="p"> + Returns <code class="ph codeph">NULL</code> if the final <code class="ph codeph">default</code> value is omitted and none of the + search expressions match the original expression. + </p> + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + <p class="p"> + The following example translates numeric day values into descriptive names: + </p> +<pre class="pre codeblock"><code>SELECT event, decode(day_of_week, 1, "Monday", 2, "Tuesday", 3, "Wednesday", + 4, "Thursday", 5, "Friday", 6, "Saturday", 7, "Sunday", "Unknown day") + FROM calendar; +</code></pre> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__if"> + <code class="ph codeph">if(boolean condition, type ifTrue, type ifFalseOrNull)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests an expression and returns a corresponding result depending on whether the result is + true, false, or <code class="ph codeph">NULL</code>. + <p class="p"> + <strong class="ph b">Return type:</strong> Same as the <code class="ph codeph">ifTrue</code> argument value + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__ifnull"> + <code class="ph codeph">ifnull(type a, type ifNull)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Alias for the <code class="ph codeph">isnull()</code> function, with the same behavior. To simplify + porting SQL with vendor extensions to Impala. + <p class="p"> + <strong class="ph b">Added in:</strong> Impala 1.3.0 + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__isfalse"> + <code class="ph codeph">isfalse(<var class="keyword varname">boolean</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if a Boolean expression is <code class="ph codeph">false</code> or not. + Returns <code class="ph codeph">true</code> if so. + If the argument is <code class="ph codeph">NULL</code>, returns <code class="ph codeph">false</code>. + Identical to <code class="ph codeph">isnottrue()</code>, except it returns the opposite value for a <code class="ph codeph">NULL</code> argument. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__isnotfalse"> + <code class="ph codeph">isnotfalse(<var class="keyword varname">boolean</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if a Boolean expression is not <code class="ph codeph">false</code> (that is, either <code class="ph codeph">true</code> or <code class="ph codeph">NULL</code>). + Returns <code class="ph codeph">true</code> if so. + If the argument is <code class="ph codeph">NULL</code>, returns <code class="ph codeph">true</code>. + Identical to <code class="ph codeph">istrue()</code>, except it returns the opposite value for a <code class="ph codeph">NULL</code> argument. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> Primarily for compatibility with code containing industry extensions to SQL. + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__isnottrue"> + <code class="ph codeph">isnottrue(<var class="keyword varname">boolean</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if a Boolean expression is not <code class="ph codeph">true</code> (that is, either <code class="ph codeph">false</code> or <code class="ph codeph">NULL</code>). + Returns <code class="ph codeph">true</code> if so. + If the argument is <code class="ph codeph">NULL</code>, returns <code class="ph codeph">true</code>. + Identical to <code class="ph codeph">isfalse()</code>, except it returns the opposite value for a <code class="ph codeph">NULL</code> argument. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__isnull"> + <code class="ph codeph">isnull(type a, type ifNull)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if an expression is <code class="ph codeph">NULL</code>, and returns the expression result value + if not. If the first argument is <code class="ph codeph">NULL</code>, returns the second argument. + <p class="p"> + <strong class="ph b">Compatibility notes:</strong> Equivalent to the <code class="ph codeph">nvl()</code> function from Oracle Database or + <code class="ph codeph">ifnull()</code> from MySQL. The <code class="ph codeph">nvl()</code> and <code class="ph codeph">ifnull()</code> + functions are also available in Impala. + </p> + <p class="p"> + <strong class="ph b">Return type:</strong> Same as the first argument value + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__istrue"> + <code class="ph codeph">istrue(<var class="keyword varname">boolean</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if a Boolean expression is <code class="ph codeph">true</code> or not. + Returns <code class="ph codeph">true</code> if so. + If the argument is <code class="ph codeph">NULL</code>, returns <code class="ph codeph">false</code>. + Identical to <code class="ph codeph">isnotfalse()</code>, except it returns the opposite value for a <code class="ph codeph">NULL</code> argument. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> Primarily for compatibility with code containing industry extensions to SQL. + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__nonnullvalue"> + <code class="ph codeph">nonnullvalue(<var class="keyword varname">expression</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if an expression (of any type) is <code class="ph codeph">NULL</code> or not. + Returns <code class="ph codeph">false</code> if so. + The converse of <code class="ph codeph">nullvalue()</code>. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> Primarily for compatibility with code containing industry extensions to SQL. + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__nullif"> + <code class="ph codeph">nullif(<var class="keyword varname">expr1</var>,<var class="keyword varname">expr2</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Returns <code class="ph codeph">NULL</code> if the two specified arguments are equal. If the specified + arguments are not equal, returns the value of <var class="keyword varname">expr1</var>. The data types of the expressions + must be compatible, according to the conversion rules from <a class="xref" href="impala_datatypes.html#datatypes">Data Types</a>. + You cannot use an expression that evaluates to <code class="ph codeph">NULL</code> for <var class="keyword varname">expr1</var>; that + way, you can distinguish a return value of <code class="ph codeph">NULL</code> from an argument value of + <code class="ph codeph">NULL</code>, which would never match <var class="keyword varname">expr2</var>. + <p class="p"> + <strong class="ph b">Usage notes:</strong> This function is effectively shorthand for a <code class="ph codeph">CASE</code> expression of + the form: + </p> +<pre class="pre codeblock"><code>CASE + WHEN <var class="keyword varname">expr1</var> = <var class="keyword varname">expr2</var> THEN NULL + ELSE <var class="keyword varname">expr1</var> +END</code></pre> + <p class="p"> + It is commonly used in division expressions, to produce a <code class="ph codeph">NULL</code> result instead of a + divide-by-zero error when the divisor is equal to zero: + </p> +<pre class="pre codeblock"><code>select 1.0 / nullif(c1,0) as reciprocal from t1;</code></pre> + <p class="p"> + You might also use it for compatibility with other database systems that support the same + <code class="ph codeph">NULLIF()</code> function. + </p> + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> Impala 1.3.0 + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__nullifzero"> + <code class="ph codeph">nullifzero(<var class="keyword varname">numeric_expr</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Returns <code class="ph codeph">NULL</code> if the numeric expression evaluates to 0, otherwise returns + the result of the expression. + <p class="p"> + <strong class="ph b">Usage notes:</strong> Used to avoid error conditions such as divide-by-zero in numeric calculations. + Serves as shorthand for a more elaborate <code class="ph codeph">CASE</code> expression, to simplify porting SQL with + vendor extensions to Impala. + </p> + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> Impala 1.3.0 + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__nullvalue"> + <code class="ph codeph">nullvalue(<var class="keyword varname">expression</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Tests if an expression (of any type) is <code class="ph codeph">NULL</code> or not. + Returns <code class="ph codeph">true</code> if so. + The converse of <code class="ph codeph">nonnullvalue()</code>. + <p class="p"> + <strong class="ph b">Return type:</strong> <code class="ph codeph">BOOLEAN</code> + </p> + <p class="p"> + <strong class="ph b">Usage notes:</strong> Primarily for compatibility with code containing industry extensions to SQL. + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.2.0</span> + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__nvl"> + <code class="ph codeph">nvl(type a, type ifNull)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Alias for the <code class="ph codeph">isnull()</code> function. Tests if an expression is + <code class="ph codeph">NULL</code>, and returns the expression result value if not. If the first argument is + <code class="ph codeph">NULL</code>, returns the second argument. Equivalent to the <code class="ph codeph">nvl()</code> function + from Oracle Database or <code class="ph codeph">ifnull()</code> from MySQL. + <p class="p"> + <strong class="ph b">Return type:</strong> Same as the first argument value + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> Impala 1.1 + </p> + </dd> + + + + + + <dt class="dt dlterm" id="conditional_functions__zeroifnull"> + <code class="ph codeph">zeroifnull(<var class="keyword varname">numeric_expr</var>)</code> + </dt> + + <dd class="dd"> + + <strong class="ph b">Purpose:</strong> Returns 0 if the numeric expression evaluates to <code class="ph codeph">NULL</code>, otherwise returns + the result of the expression. + <p class="p"> + <strong class="ph b">Usage notes:</strong> Used to avoid unexpected results due to unexpected propagation of + <code class="ph codeph">NULL</code> values in numeric calculations. Serves as shorthand for a more elaborate + <code class="ph codeph">CASE</code> expression, to simplify porting SQL with vendor extensions to Impala. + </p> + <p class="p"> + <strong class="ph b">Return type:</strong> same as the initial argument value, except that integer values are promoted to + <code class="ph codeph">BIGINT</code> and floating-point values are promoted to <code class="ph codeph">DOUBLE</code>; use + <code class="ph codeph">CAST()</code> when inserting into a smaller numeric column + </p> + <p class="p"> + <strong class="ph b">Added in:</strong> Impala 1.3.0 + </p> + </dd> + + + </dl> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_functions.html">Impala Built-In Functions</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_config.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_config.html b/docs/build/html/topics/impala_config.html new file mode 100644 index 0000000..6950647 --- /dev/null +++ b/docs/build/html/topics/impala_config.html @@ -0,0 +1,48 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_config_performance.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_odbc.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_jdbc.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="config"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Managing Impala</title></head><body id="config"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Managing Impala</h1> + + + <div class="body conbody"> + + <p class="p"> + This section explains how to configure Impala to accept connections from applications that use popular + programming APIs: + </p> + + <ul class="ul"> + <li class="li"> + <a class="xref" href="impala_config_performance.html#config_performance">Post-Installation Configuration for Impala</a> + </li> + + <li class="li"> + <a class="xref" href="impala_odbc.html#impala_odbc">Configuring Impala to Work with ODBC</a> + </li> + + <li class="li"> + <a class="xref" href="impala_jdbc.html#impala_jdbc">Configuring Impala to Work with JDBC</a> + </li> + </ul> + + <p class="p"> + This type of configuration is especially useful when using Impala in combination with Business Intelligence + tools, which use these standard interfaces to query different kinds of database and Big Data systems. + </p> + + <p class="p"> + You can also configure these other aspects of Impala: + </p> + + <ul class="ul"> + <li class="li"> + <a class="xref" href="impala_security.html#security">Impala Security</a> + </li> + + <li class="li"> + <a class="xref" href="impala_config_options.html#config_options">Modifying Impala Startup Options</a> + </li> + </ul> + </div> +<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_config_performance.html">Post-Installation Configuration for Impala</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_odbc.html">Configuring Impala to Work with ODBC</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_jdbc.html">Configuring Impala to Work with JDBC</a></strong><br></li></ul></nav></article></main></body></html> \ No newline at end of file
