http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_security_guidelines.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_security_guidelines.html b/docs/build/html/topics/impala_security_guidelines.html new file mode 100644 index 0000000..4b1a738 --- /dev/null +++ b/docs/build/html/topics/impala_security_guidelines.html @@ -0,0 +1,99 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_security.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="security_guidelines"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Security Guidelines for Impala</title></head><body id="security_guidelines"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Security Guidelines for Impala</h1> + + + <div class="body conbody"> + + <p class="p"> + The following are the major steps to harden a cluster running Impala against accidents and mistakes, or + malicious attackers trying to access sensitive data: + </p> + + <ul class="ul"> + <li class="li"> + <p class="p"> + Secure the <code class="ph codeph">root</code> account. The <code class="ph codeph">root</code> user can tamper with the + <span class="keyword cmdname">impalad</span> daemon, read and write the data files in HDFS, log into other user accounts, and + access other system services that are beyond the control of Impala. + </p> + </li> + + <li class="li"> + <p class="p"> + Restrict membership in the <code class="ph codeph">sudoers</code> list (in the <span class="ph filepath">/etc/sudoers</span> file). + The users who can run the <code class="ph codeph">sudo</code> command can do many of the same things as the + <code class="ph codeph">root</code> user. + </p> + </li> + + <li class="li"> + <p class="p"> + Ensure the Hadoop ownership and permissions for Impala data files are restricted. + </p> + </li> + + <li class="li"> + <p class="p"> + Ensure the Hadoop ownership and permissions for Impala log files are restricted. + </p> + </li> + + <li class="li"> + <p class="p"> + Ensure that the Impala web UI (available by default on port 25000 on each Impala node) is + password-protected. See <a class="xref" href="impala_webui.html#webui">Impala Web User Interface for Debugging</a> for details. + </p> + </li> + + <li class="li"> + <p class="p"> + Create a policy file that specifies which Impala privileges are available to users in particular Hadoop + groups (which by default map to Linux OS groups). Create the associated Linux groups using the + <span class="keyword cmdname">groupadd</span> command if necessary. + </p> + </li> + + <li class="li"> + <p class="p"> + The Impala authorization feature makes use of the HDFS file ownership and permissions mechanism; for + background information, see the + <a class="xref" href="https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html"; target="_blank">HDFS Permissions Guide</a>. + Set up users and assign them to groups at the OS level, corresponding to the + different categories of users with different access levels for various databases, tables, and HDFS + locations (URIs). Create the associated Linux users using the <span class="keyword cmdname">useradd</span> command if + necessary, and add them to the appropriate groups with the <span class="keyword cmdname">usermod</span> command. + </p> + </li> + + <li class="li"> + <p class="p"> + Design your databases, tables, and views with database and table structure to allow policy rules to specify + simple, consistent rules. For example, if all tables related to an application are inside a single + database, you can assign privileges for that database and use the <code class="ph codeph">*</code> wildcard for the table + name. If you are creating views with different privileges than the underlying base tables, you might put + the views in a separate database so that you can use the <code class="ph codeph">*</code> wildcard for the database + containing the base tables, while specifying the precise names of the individual views. (For specifying + table or database names, you either specify the exact name or <code class="ph codeph">*</code> to mean all the databases + on a server, or all the tables and views in a database.) + </p> + </li> + + <li class="li"> + <p class="p"> + Enable authorization by running the <code class="ph codeph">impalad</code> daemons with the <code class="ph codeph">-server_name</code> + and <code class="ph codeph">-authorization_policy_file</code> options on all nodes. (The authorization feature does not + apply to the <span class="keyword cmdname">statestored</span> daemon, which has no access to schema objects or data files.) + </p> + </li> + + <li class="li"> + <p class="p"> + Set up authentication using Kerberos, to make sure users really are who they say they are. + </p> + </li> + </ul> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_security.html">Impala Security</a></div></div></nav></article></main></body></html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_security_install.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_security_install.html b/docs/build/html/topics/impala_security_install.html new file mode 100644 index 0000000..f9724ef --- /dev/null +++ b/docs/build/html/topics/impala_security_install.html @@ -0,0 +1,17 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_security.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="security_install"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Installation Considerations for Impala Security</title></head><body id="security_install"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Installation Considerations for Impala Security</h1> + + + <div class="body conbody"> + + <p class="p"> + Impala 1.1 comes set up with all the software and settings needed to enable security when you run the + <span class="keyword cmdname">impalad</span> daemon with the new security-related options (<code class="ph codeph">-server_name</code> and + <code class="ph codeph">-authorization_policy_file</code>). You do not need to change any environment variables or install + any additional JAR files. + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_security.html">Impala Security</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_security_metastore.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_security_metastore.html b/docs/build/html/topics/impala_security_metastore.html new file mode 100644 index 0000000..cc852ad --- /dev/null +++ b/docs/build/html/topics/impala_security_metastore.html @@ -0,0 +1,30 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_security.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="security_metastore"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Securing the Hive Metastore Database</title></head><body id="security_metastore"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Securing the Hive Metastore Database</h1> + + + <div class="body conbody"> + + + + <p class="p"> + It is important to secure the Hive metastore, so that users cannot access the names or other information + about databases and tables the through the Hive client or by querying the metastore database. Do this by + turning on Hive metastore security, using the instructions in + <span class="xref">the documentation for your Apache Hadoop distribution</span> for securing different Hive components: + </p> + + <ul class="ul"> + <li class="li"> + Secure the Hive Metastore. + </li> + + <li class="li"> + In addition, allow access to the metastore only from the HiveServer2 server, and then disable local access + to the HiveServer2 server. + </li> + </ul> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_security.html">Impala Security</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_security_webui.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_security_webui.html b/docs/build/html/topics/impala_security_webui.html new file mode 100644 index 0000000..6286012 --- /dev/null +++ b/docs/build/html/topics/impala_security_webui.html @@ -0,0 +1,57 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_security.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="security_webui"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Securing the Impala Web User Interface</title></head><body id="security_webui"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">Securing the Impala Web User Interface</h1> + + + <div class="body conbody"> + + <p class="p"> + The instructions in this section presume you are familiar with the + <a class="xref" href="http://en.wikipedia.org/wiki/.htpasswd"; target="_blank"> + <span class="ph filepath">.htpasswd</span> mechanism</a> commonly used to password-protect pages on web servers. + </p> + + <p class="p"> + Password-protect the Impala web UI that listens on port 25000 by default. Set up a + <span class="ph filepath">.htpasswd</span> file in the <code class="ph codeph">$IMPALA_HOME</code> directory, or start both the + <span class="keyword cmdname">impalad</span> and <span class="keyword cmdname">statestored</span> daemons with the + <code class="ph codeph">--webserver_password_file</code> option to specify a different location (including the filename). + </p> + + <p class="p"> + This file should only be readable by the Impala process and machine administrators, because it contains + (hashed) versions of passwords. The username / password pairs are not derived from Unix usernames, Kerberos + users, or any other system. The <code class="ph codeph">domain</code> field in the password file must match the domain + supplied to Impala by the new command-line option <code class="ph codeph">--webserver_authentication_domain</code>. The + default is <code class="ph codeph">mydomain.com</code>. + + </p> + + <p class="p"> + Impala also supports using HTTPS for secure web traffic. To do so, set + <code class="ph codeph">--webserver_certificate_file</code> to refer to a valid <code class="ph codeph">.pem</code> TLS/SSL certificate file. + Impala will automatically start using HTTPS once the TLS/SSL certificate has been read and validated. A + <code class="ph codeph">.pem</code> file is basically a private key, followed by a signed TLS/SSL certificate; make sure to + concatenate both parts when constructing the <code class="ph codeph">.pem</code> file. + + </p> + + <p class="p"> + If Impala cannot find or parse the <code class="ph codeph">.pem</code> file, it prints an error message and quits. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + If the private key is encrypted using a passphrase, Impala will ask for that passphrase on startup, which + is not useful for a large cluster. In that case, remove the passphrase and make the <code class="ph codeph">.pem</code> + file readable only by Impala and administrators. + </p> + <p class="p"> + When you turn on TLS/SSL for the Impala web UI, the associated URLs change from <code class="ph codeph">http://</code> + prefixes to <code class="ph codeph">https://</code>. Adjust any bookmarks or application code that refers to those URLs. + </p> + </div> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_security.html">Impala Security</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_select.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_select.html b/docs/build/html/topics/impala_select.html new file mode 100644 index 0000000..7a12c42 --- /dev/null +++ b/docs/build/html/topics/impala_select.html @@ -0,0 +1,227 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_joins.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_order_by.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_group_by.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_having.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_limit.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_offset.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_union.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_subqueries.html"><meta name="DC.Relation" scheme="U RI" content="../topics/impala_with.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_distinct.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_hints.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="select"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>SELECT Statement</title></head><body id="select"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">SELECT Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + The <code class="ph codeph">SELECT</code> statement performs queries, retrieving data from one or more tables and producing + result sets consisting of rows and columns. + </p> + + <p class="p"> + The Impala <code class="ph codeph"><a class="xref" href="impala_insert.html#insert">INSERT</a></code> statement also typically ends + with a <code class="ph codeph">SELECT</code> statement, to define data to copy from one table to another. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>[WITH <em class="ph i">name</em> AS (<em class="ph i">select_expression</em>) [, ...] ] +SELECT + [ALL | DISTINCT] + [STRAIGHT_JOIN] + <em class="ph i">expression</em> [, <em class="ph i">expression</em> ...] +FROM <em class="ph i">table_reference</em> [, <em class="ph i">table_reference</em> ...] +[[FULL | [LEFT | RIGHT] INNER | [LEFT | RIGHT] OUTER | [LEFT | RIGHT] SEMI | [LEFT | RIGHT] ANTI | CROSS] + JOIN <em class="ph i">table_reference</em> + [ON <em class="ph i">join_equality_clauses</em> | USING (<var class="keyword varname">col1</var>[, <var class="keyword varname">col2</var> ...]] ... +WHERE <em class="ph i">conditions</em> +GROUP BY { <em class="ph i">column</em> | <em class="ph i">expression</em> [ASC | DESC] [NULLS FIRST | NULLS LAST] [, ...] } +HAVING <code class="ph codeph">conditions</code> +GROUP BY { <em class="ph i">column</em> | <em class="ph i">expression</em> [ASC | DESC] [, ...] } +LIMIT <em class="ph i">expression</em> [OFFSET <em class="ph i">expression</em>] +[UNION [ALL] <em class="ph i">select_statement</em>] ...] +</code></pre> + + <p class="p"> + Impala <code class="ph codeph">SELECT</code> queries support: + </p> + + <ul class="ul"> + <li class="li"> + SQL scalar data types: <code class="ph codeph"><a class="xref" href="impala_boolean.html#boolean">BOOLEAN</a></code>, + <code class="ph codeph"><a class="xref" href="impala_tinyint.html#tinyint">TINYINT</a></code>, + <code class="ph codeph"><a class="xref" href="impala_smallint.html#smallint">SMALLINT</a></code>, + <code class="ph codeph"><a class="xref" href="impala_int.html#int">INT</a></code>, + <code class="ph codeph"><a class="xref" href="impala_bigint.html#bigint">BIGINT</a></code>, + <code class="ph codeph"><a class="xref" href="impala_decimal.html#decimal">DECIMAL</a></code> + <code class="ph codeph"><a class="xref" href="impala_float.html#float">FLOAT</a></code>, + <code class="ph codeph"><a class="xref" href="impala_double.html#double">DOUBLE</a></code>, + <code class="ph codeph"><a class="xref" href="impala_timestamp.html#timestamp">TIMESTAMP</a></code>, + <code class="ph codeph"><a class="xref" href="impala_string.html#string">STRING</a></code>, + <code class="ph codeph"><a class="xref" href="impala_varchar.html#varchar">VARCHAR</a></code>, + <code class="ph codeph"><a class="xref" href="impala_char.html#char">CHAR</a></code>. + </li> + + + <li class="li"> + The complex data types <code class="ph codeph">ARRAY</code>, <code class="ph codeph">STRUCT</code>, and <code class="ph codeph">MAP</code>, + are available in <span class="keyword">Impala 2.3</span> and higher. + Queries involving these types typically involve special qualified names + using dot notation for referring to the complex column fields, + and join clauses for bringing the complex columns into the result set. + See <a class="xref" href="impala_complex_types.html#complex_types">Complex Types (Impala 2.3 or higher only)</a> for details. + </li> + + <li class="li"> + An optional <a class="xref" href="impala_with.html#with"><code class="ph codeph">WITH</code> clause</a> before the + <code class="ph codeph">SELECT</code> keyword, to define a subquery whose name or column names can be referenced from + later in the main query. This clause lets you abstract repeated clauses, such as aggregation functions, + that are referenced multiple times in the same query. + </li> + + <li class="li"> + By default, one <code class="ph codeph">DISTINCT</code> clause per query. See <a class="xref" href="impala_distinct.html#distinct">DISTINCT Operator</a> + for details. See <a class="xref" href="impala_appx_count_distinct.html#appx_count_distinct">APPX_COUNT_DISTINCT Query Option (Impala 2.0 or higher only)</a> for a query option to + allow multiple <code class="ph codeph">COUNT(DISTINCT)</code> impressions in the same query. + </li> + + <li class="li"> + Subqueries in a <code class="ph codeph">FROM</code> clause. In <span class="keyword">Impala 2.0</span> and higher, + subqueries can also go in the <code class="ph codeph">WHERE</code> clause, for example with the + <code class="ph codeph">IN()</code>, <code class="ph codeph">EXISTS</code>, and <code class="ph codeph">NOT EXISTS</code> operators. + </li> + + <li class="li"> + <code class="ph codeph">WHERE</code>, <code class="ph codeph">GROUP BY</code>, <code class="ph codeph">HAVING</code> clauses. + </li> + + <li class="li"> + <code class="ph codeph"><a class="xref" href="impala_order_by.html#order_by">ORDER BY</a></code>. Prior to Impala 1.4.0, Impala + required that queries using an <code class="ph codeph">ORDER BY</code> clause also include a + <code class="ph codeph"><a class="xref" href="impala_limit.html#limit">LIMIT</a></code> clause. In Impala 1.4.0 and higher, this + restriction is lifted; sort operations that would exceed the Impala memory limit automatically use a + temporary disk work area to perform the sort. + </li> + + <li class="li"> + <p class="p"> + Impala supports a wide variety of <code class="ph codeph">JOIN</code> clauses. Left, right, semi, full, and outer joins + are supported in all Impala versions. The <code class="ph codeph">CROSS JOIN</code> operator is available in Impala 1.2.2 + and higher. During performance tuning, you can override the reordering of join clauses that Impala does + internally by including the keyword <code class="ph codeph">STRAIGHT_JOIN</code> immediately after the + <code class="ph codeph">SELECT</code> keyword + </p> + <p class="p"> + See <a class="xref" href="impala_joins.html#joins">Joins in Impala SELECT Statements</a> for details and examples of join queries. + </p> + </li> + + <li class="li"> + <code class="ph codeph">UNION ALL</code>. + </li> + + <li class="li"> + <code class="ph codeph">LIMIT</code>. + </li> + + <li class="li"> + External tables. + </li> + + <li class="li"> + Relational operators such as greater than, less than, or equal to. + </li> + + <li class="li"> + Arithmetic operators such as addition or subtraction. + </li> + + <li class="li"> + Logical/Boolean operators <code class="ph codeph">AND</code>, <code class="ph codeph">OR</code>, and <code class="ph codeph">NOT</code>. Impala does + not support the corresponding symbols <code class="ph codeph">&&</code>, <code class="ph codeph">||</code>, and + <code class="ph codeph">!</code>. + </li> + + <li class="li"> + Common SQL built-in functions such as <code class="ph codeph">COUNT</code>, <code class="ph codeph">SUM</code>, <code class="ph codeph">CAST</code>, + <code class="ph codeph">LIKE</code>, <code class="ph codeph">IN</code>, <code class="ph codeph">BETWEEN</code>, and <code class="ph codeph">COALESCE</code>. Impala + specifically supports built-ins described in <a class="xref" href="impala_functions.html#builtins">Impala Built-In Functions</a>. + </li> + </ul> + + <p class="p"> + Impala queries ignore files with extensions commonly used for temporary work files by Hadoop tools. Any + files with extensions <code class="ph codeph">.tmp</code> or <code class="ph codeph">.copying</code> are not considered part of the + Impala table. The suffix matching is case-insensitive, so for example Impala ignores both + <code class="ph codeph">.copying</code> and <code class="ph codeph">.COPYING</code> suffixes. + </p> + + <p class="p"> + <strong class="ph b">Security considerations:</strong> + </p> + <p class="p"> + If these statements in your environment contain sensitive literal values such as credit card numbers or tax + identifiers, Impala can redact this sensitive information when displaying the statements in log files and + other administrative contexts. See <span class="xref">the documentation for your Apache Hadoop distribution</span> for details. + </p> + + <p class="p"> + <strong class="ph b">Amazon S3 considerations:</strong> + </p> + <p class="p"> + In <span class="keyword">Impala 2.6</span> and higher, Impala queries are optimized for files stored in Amazon S3. + For Impala tables that use the file formats Parquet, RCFile, SequenceFile, + Avro, and uncompressed text, the setting <code class="ph codeph">fs.s3a.block.size</code> + in the <span class="ph filepath">core-site.xml</span> configuration file determines + how Impala divides the I/O work of reading the data files. This configuration + setting is specified in bytes. By default, this + value is 33554432 (32 MB), meaning that Impala parallelizes S3 read operations on the files + as if they were made up of 32 MB blocks. For example, if your S3 queries primarily access + Parquet files written by MapReduce or Hive, increase <code class="ph codeph">fs.s3a.block.size</code> + to 134217728 (128 MB) to match the row group size of those files. If most S3 queries involve + Parquet files written by Impala, increase <code class="ph codeph">fs.s3a.block.size</code> + to 268435456 (256 MB) to match the row group size produced by Impala. + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Can be cancelled. To cancel this statement, use Ctrl-C from the + <span class="keyword cmdname">impala-shell</span> interpreter, the <span class="ph uicontrol">Cancel</span> button from the + <span class="ph uicontrol">Watch</span> page in Hue, or <span class="ph uicontrol">Cancel</span> from the list of + in-flight queries (for a particular node) on the <span class="ph uicontrol">Queries</span> tab in the Impala web UI + (port 25000). + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, must have read + permissions for the files in all applicable directories in all source tables, + and read and execute permissions for the relevant data directories. + (A <code class="ph codeph">SELECT</code> operation could read files from multiple different HDFS directories + if the source table is partitioned.) + If a query attempts to read a data file and is unable to because of an HDFS permission error, + the query halts and does not return any further results. + </p> + + <p class="p toc"></p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + The <code class="ph codeph">SELECT</code> syntax is so extensive that it forms its own category of statements: queries. The + other major classifications of SQL statements are data definition language (see + <a class="xref" href="impala_ddl.html#ddl">DDL Statements</a>) and data manipulation language (see <a class="xref" href="impala_dml.html#dml">DML Statements</a>). + </p> + + <p class="p"> + Because the focus of Impala is on fast queries with interactive response times over huge data sets, query + performance and scalability are important considerations. See + <a class="xref" href="impala_performance.html#performance">Tuning Impala for Performance</a> and <a class="xref" href="impala_scalability.html#scalability">Scalability Considerations for Impala</a> for + details. + </p> + </div> + + +<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_joins.html">Joins in Impala SELECT Statements</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_order_by.html">ORDER BY Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_group_by.html">GROUP BY Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_having.html">HAVING Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_limit.html">LIMIT Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_offset.html">OFFSET Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_union.html">UNION Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_subqueries.html">Subqueries in Impala SELECT Statements</a></strong>< br></li><li class="link ulchildlink"><strong><a href="../topics/impala_with.html">WITH Clause</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_distinct.html">DISTINCT Operator</a></strong><br></li><li class="link ulchildlink"><strong><a href="../topics/impala_hints.html">Query Hints in Impala SELECT Statements</a></strong><br></li></ul><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_seqfile.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_seqfile.html b/docs/build/html/topics/impala_seqfile.html new file mode 100644 index 0000000..53a0eaf --- /dev/null +++ b/docs/build/html/topics/impala_seqfile.html @@ -0,0 +1,240 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_file_formats.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="seqfile"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>Using the SequenceFile File Format with Impala Tables</title></head><body id="seqfile"><main role="main"><article role="article" aria-labelledby="seqfile__sequencefile"> + + <h1 class="title topictitle1" id="seqfile__sequencefile">Using the SequenceFile File Format with Impala Tables</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Impala supports using SequenceFile data files. + </p> + + <table class="table"><caption><span class="table--title-label">Table 1. </span><span class="title">SequenceFile Format Support in Impala</span></caption><colgroup><col style="width:10%"><col style="width:10%"><col style="width:20%"><col style="width:30%"><col style="width:30%"></colgroup><thead class="thead"> + <tr class="row"> + <th class="entry nocellnorowborder" id="seqfile__entry__1"> + File Type + </th> + <th class="entry nocellnorowborder" id="seqfile__entry__2"> + Format + </th> + <th class="entry nocellnorowborder" id="seqfile__entry__3"> + Compression Codecs + </th> + <th class="entry nocellnorowborder" id="seqfile__entry__4"> + Impala Can CREATE? + </th> + <th class="entry nocellnorowborder" id="seqfile__entry__5"> + Impala Can INSERT? + </th> + </tr> + </thead><tbody class="tbody"> + <tr class="row"> + <td class="entry nocellnorowborder" headers="seqfile__entry__1 "> + <a class="xref" href="impala_seqfile.html#seqfile">SequenceFile</a> + </td> + <td class="entry nocellnorowborder" headers="seqfile__entry__2 "> + Structured + </td> + <td class="entry nocellnorowborder" headers="seqfile__entry__3 "> + Snappy, gzip, deflate, bzip2 + </td> + <td class="entry nocellnorowborder" headers="seqfile__entry__4 ">Yes.</td> + <td class="entry nocellnorowborder" headers="seqfile__entry__5 "> + No. Import data by using <code class="ph codeph">LOAD DATA</code> on data files already in the right format, or use + <code class="ph codeph">INSERT</code> in Hive followed by <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> in Impala. + </td> + + </tr> + </tbody></table> + + <p class="p toc inpage"></p> + </div> + + <nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_file_formats.html">How Impala Works with Hadoop File Formats</a></div></div></nav><article class="topic concept nested1" aria-labelledby="ariaid-title2" id="seqfile__seqfile_create"> + + <h2 class="title topictitle2" id="ariaid-title2">Creating SequenceFile Tables and Loading Data</h2> + + + <div class="body conbody"> + + <p class="p"> + If you do not have an existing data file to use, begin by creating one in the appropriate format. + </p> + + <p class="p"> + <strong class="ph b">To create a SequenceFile table:</strong> + </p> + + <p class="p"> + In the <code class="ph codeph">impala-shell</code> interpreter, issue a command similar to: + </p> + +<pre class="pre codeblock"><code>create table sequencefile_table (<var class="keyword varname">column_specs</var>) stored as sequencefile;</code></pre> + + <p class="p"> + Because Impala can query some kinds of tables that it cannot currently write to, after creating tables of + certain file formats, you might use the Hive shell to load the data. See + <a class="xref" href="impala_file_formats.html#file_formats">How Impala Works with Hadoop File Formats</a> for details. After loading data into a table through + Hive or other mechanism outside of Impala, issue a <code class="ph codeph">REFRESH <var class="keyword varname">table_name</var></code> + statement the next time you connect to the Impala node, before querying the table, to make Impala recognize + the new data. + </p> + + <p class="p"> + For example, here is how you might create some SequenceFile tables in Impala (by specifying the columns + explicitly, or cloning the structure of another table), load data through Hive, and query them through + Impala: + </p> + +<pre class="pre codeblock"><code>$ impala-shell -i localhost +[localhost:21000] > create table seqfile_table (x int) stored as sequencefile; +[localhost:21000] > create table seqfile_clone like some_other_table stored as sequencefile; +[localhost:21000] > quit; + +$ hive +hive> insert into table seqfile_table select x from some_other_table; +3 Rows loaded to seqfile_table +Time taken: 19.047 seconds +hive> quit; + +$ impala-shell -i localhost +[localhost:21000] > select * from seqfile_table; +Returned 0 row(s) in 0.23s +[localhost:21000] > -- Make Impala recognize the data loaded through Hive; +[localhost:21000] > refresh seqfile_table; +[localhost:21000] > select * from seqfile_table; ++---+ +| x | ++---+ +| 1 | +| 2 | +| 3 | ++---+ +Returned 3 row(s) in 0.23s</code></pre> + + <p class="p"> + <strong class="ph b">Complex type considerations:</strong> + Although you can create tables in this file format using + the complex types (<code class="ph codeph">ARRAY</code>, <code class="ph codeph">STRUCT</code>, + and <code class="ph codeph">MAP</code>) available in <span class="keyword">Impala 2.3</span> and higher, + currently, Impala can query these types only in Parquet tables. + <span class="ph"> + The one exception to the preceding rule is <code class="ph codeph">COUNT(*)</code> queries on RCFile tables that include complex types. + Such queries are allowed in <span class="keyword">Impala 2.6</span> and higher. + </span> + </p> + + </div> + </article> + + <article class="topic concept nested1" aria-labelledby="ariaid-title3" id="seqfile__seqfile_compression"> + + <h2 class="title topictitle2" id="ariaid-title3">Enabling Compression for SequenceFile Tables</h2> + + + <div class="body conbody"> + + <p class="p"> + + You may want to enable compression on existing tables. Enabling compression provides performance gains in + most cases and is supported for SequenceFile tables. For example, to enable Snappy compression, you would + specify the following additional settings when loading data through the Hive shell: + </p> + +<pre class="pre codeblock"><code>hive> SET hive.exec.compress.output=true; +hive> SET mapred.max.split.size=256000000; +hive> SET mapred.output.compression.type=BLOCK; +hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec; +hive> insert overwrite table <var class="keyword varname">new_table</var> select * from <var class="keyword varname">old_table</var>;</code></pre> + + <p class="p"> + If you are converting partitioned tables, you must complete additional steps. In such a case, specify + additional settings similar to the following: + </p> + +<pre class="pre codeblock"><code>hive> create table <var class="keyword varname">new_table</var> (<var class="keyword varname">your_cols</var>) partitioned by (<var class="keyword varname">partition_cols</var>) stored as <var class="keyword varname">new_format</var>; +hive> SET hive.exec.dynamic.partition.mode=nonstrict; +hive> SET hive.exec.dynamic.partition=true; +hive> insert overwrite table <var class="keyword varname">new_table</var> partition(<var class="keyword varname">comma_separated_partition_cols</var>) select * from <var class="keyword varname">old_table</var>;</code></pre> + + <p class="p"> + Remember that Hive does not require that you specify a source format for it. Consider the case of + converting a table with two partition columns called <code class="ph codeph">year</code> and <code class="ph codeph">month</code> to a + Snappy compressed SequenceFile. Combining the components outlined previously to complete this table + conversion, you would specify settings similar to the following: + </p> + +<pre class="pre codeblock"><code>hive> create table TBL_SEQ (int_col int, string_col string) STORED AS SEQUENCEFILE; +hive> SET hive.exec.compress.output=true; +hive> SET mapred.max.split.size=256000000; +hive> SET mapred.output.compression.type=BLOCK; +hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec; +hive> SET hive.exec.dynamic.partition.mode=nonstrict; +hive> SET hive.exec.dynamic.partition=true; +hive> INSERT OVERWRITE TABLE tbl_seq SELECT * FROM tbl;</code></pre> + + <p class="p"> + To complete a similar process for a table that includes partitions, you would specify settings similar to + the following: + </p> + +<pre class="pre codeblock"><code>hive> CREATE TABLE tbl_seq (int_col INT, string_col STRING) PARTITIONED BY (year INT) STORED AS SEQUENCEFILE; +hive> SET hive.exec.compress.output=true; +hive> SET mapred.max.split.size=256000000; +hive> SET mapred.output.compression.type=BLOCK; +hive> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec; +hive> SET hive.exec.dynamic.partition.mode=nonstrict; +hive> SET hive.exec.dynamic.partition=true; +hive> INSERT OVERWRITE TABLE tbl_seq PARTITION(year) SELECT * FROM tbl;</code></pre> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + The compression type is specified in the following command: + </p> +<pre class="pre codeblock"><code>SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;</code></pre> + <p class="p"> + You could elect to specify alternative codecs such as <code class="ph codeph">GzipCodec</code> here. + </p> + </div> + </div> + </article> + + + + <article class="topic concept nested1" aria-labelledby="ariaid-title4" id="seqfile__seqfile_performance"> + + <h2 class="title topictitle2" id="ariaid-title4">Query Performance for Impala SequenceFile Tables</h2> + + <div class="body conbody"> + + <p class="p"> + In general, expect query performance with SequenceFile tables to be + faster than with tables using text data, but slower than with + Parquet tables. See <a class="xref" href="impala_parquet.html#parquet">Using the Parquet File Format with Impala Tables</a> + for information about using the Parquet file format for + high-performance analytic queries. + </p> + + <p class="p"> + In <span class="keyword">Impala 2.6</span> and higher, Impala queries are optimized for files stored in Amazon S3. + For Impala tables that use the file formats Parquet, RCFile, SequenceFile, + Avro, and uncompressed text, the setting <code class="ph codeph">fs.s3a.block.size</code> + in the <span class="ph filepath">core-site.xml</span> configuration file determines + how Impala divides the I/O work of reading the data files. This configuration + setting is specified in bytes. By default, this + value is 33554432 (32 MB), meaning that Impala parallelizes S3 read operations on the files + as if they were made up of 32 MB blocks. For example, if your S3 queries primarily access + Parquet files written by MapReduce or Hive, increase <code class="ph codeph">fs.s3a.block.size</code> + to 134217728 (128 MB) to match the row group size of those files. If most S3 queries involve + Parquet files written by Impala, increase <code class="ph codeph">fs.s3a.block.size</code> + to 268435456 (256 MB) to match the row group size produced by Impala. + </p> + + </div> + </article> + +</article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_set.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_set.html b/docs/build/html/topics/impala_set.html new file mode 100644 index 0000000..b16ff7b --- /dev/null +++ b/docs/build/html/topics/impala_set.html @@ -0,0 +1,200 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="set"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>SET Statement</title></head><body id="set"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">SET Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Specifies values for query options that control the runtime behavior of other statements within the same + session. + </p> + + <p class="p"> + In <span class="keyword">Impala 2.5</span> and higher, <code class="ph codeph">SET</code> also defines user-specified substitution variables for + the <span class="keyword cmdname">impala-shell</span> interpreter. This feature uses the <code class="ph codeph">SET</code> command + built into <span class="keyword cmdname">impala-shell</span> instead of the SQL <code class="ph codeph">SET</code> statement. + Therefore the substitution mechanism only works with queries processed by <span class="keyword cmdname">impala-shell</span>, + not with queries submitted through JDBC or ODBC. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>SET [<var class="keyword varname">query_option</var>=<var class="keyword varname">option_value</var>] +</code></pre> + + <p class="p"> + <code class="ph codeph">SET</code> with no arguments returns a result set consisting of all available query options and + their current values. + </p> + + <p class="p"> + The query option name and any string argument values are case-insensitive. + </p> + + <p class="p"> + Each query option has a specific allowed notation for its arguments. Boolean options can be enabled and + disabled by assigning values of either <code class="ph codeph">true</code> and <code class="ph codeph">false</code>, or + <code class="ph codeph">1</code> and <code class="ph codeph">0</code>. Some numeric options accept a final character signifying the unit, + such as <code class="ph codeph">2g</code> for 2 gigabytes or <code class="ph codeph">100m</code> for 100 megabytes. See + <a class="xref" href="impala_query_options.html#query_options">Query Options for the SET Statement</a> for the details of each query option. + </p> + + <p class="p"> + <strong class="ph b">User-specified substitution variables:</strong> + </p> + + <p class="p"> + In <span class="keyword">Impala 2.5</span> and higher, you can specify your own names and string substitution values + within the <span class="keyword cmdname">impala-shell</span> interpreter. Once a substitution variable is set up, + its value is inserted into any SQL statement in that same <span class="keyword cmdname">impala-shell</span> session + that contains the notation <code class="ph codeph">${var:<var class="keyword varname">varname</var>}</code>. + Using <code class="ph codeph">SET</code> in an interactive <span class="keyword cmdname">impala-shell</span> session overrides + any value for that same variable passed in through the <code class="ph codeph">--var=<var class="keyword varname">varname</var>=<var class="keyword varname">value</var></code> + command-line option. + </p> + + <p class="p"> + For example, to set up some default parameters for report queries, but then override those default + within an <span class="keyword cmdname">impala-shell</span> session, you might issue commands and statements such as + the following: + </p> + +<pre class="pre codeblock"><code> +-- Initial setup for this example. +create table staging_table (s string); +insert into staging_table values ('foo'), ('bar'), ('bletch'); + +create table production_table (s string); +insert into production_table values ('North America'), ('EMEA'), ('Asia'); +quit; + +-- Start impala-shell with user-specified substitution variables, +-- run a query, then override the variables with SET and run the query again. +$ impala-shell --var=table_name=staging_table --var=cutoff=2 +... <var class="keyword varname">banner message</var> ... +[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff}; +Query: select s from staging_table order by s limit 2 ++--------+ +| s | ++--------+ +| bar | +| bletch | ++--------+ +Fetched 2 row(s) in 1.06s + +[localhost:21000] > set var:table_name=production_table; +Variable TABLE_NAME set to production_table +[localhost:21000] > set var:cutoff=3; +Variable CUTOFF set to 3 + +[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff}; +Query: select s from production_table order by s limit 3 ++---------------+ +| s | ++---------------+ +| Asia | +| EMEA | +| North America | ++---------------+ +</code></pre> + + <p class="p"> + The following example shows how <code class="ph codeph">SET</code> with no parameters displays + all user-specified substitution variables, and how <code class="ph codeph">UNSET</code> removes + the substitution variable entirely: + </p> + +<pre class="pre codeblock"><code> +[localhost:21000] > set; +Query options (defaults shown in []): + ABORT_ON_DEFAULT_LIMIT_EXCEEDED: [0] + ... + V_CPU_CORES: [0] + +Shell Options + LIVE_PROGRESS: False + LIVE_SUMMARY: False + +Variables: + CUTOFF: 3 + TABLE_NAME: staging_table + +[localhost:21000] > unset var:cutoff; +Unsetting variable CUTOFF +[localhost:21000] > select s from ${var:table_name} order by s limit ${var:cutoff}; +Error: Unknown variable CUTOFF +</code></pre> + + <p class="p"> + See <a class="xref" href="impala_shell_running_commands.html">Running Commands and SQL Statements in impala-shell</a> for more examples of using the + <code class="ph codeph">--var</code>, <code class="ph codeph">SET</code>, and <code class="ph codeph">${var:<var class="keyword varname">varname</var>}</code> + substitution technique in <span class="keyword cmdname">impala-shell</span>. + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + <code class="ph codeph">MEM_LIMIT</code> is probably the most commonly used query option. You can specify a high value to + allow a resource-intensive query to complete. For testing how queries would work on memory-constrained + systems, you might specify an artificially low value. + </p> + + <p class="p"> + <strong class="ph b">Complex type considerations:</strong> + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <p class="p"> + The following example sets some numeric and some Boolean query options to control usage of memory, disk + space, and timeout periods, then runs a query whose success could depend on the options in effect: + </p> + +<pre class="pre codeblock"><code>set mem_limit=64g; +set DISABLE_UNSAFE_SPILLS=true; +set parquet_file_size=400m; +set RESERVATION_REQUEST_TIMEOUT=900000; +insert overwrite parquet_table select c1, c2, count(c3) from text_table group by c1, c2, c3; +</code></pre> + + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.0.0</span> + </p> + + <p class="p"> + <code class="ph codeph">SET</code> has always been available as an <span class="keyword cmdname">impala-shell</span> command. Promoting it to + a SQL statement lets you use this feature in client applications through the JDBC and ODBC APIs. + </p> + + + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> This statement does not touch any HDFS files or directories, + therefore no HDFS permissions are required. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + See <a class="xref" href="impala_query_options.html#query_options">Query Options for the SET Statement</a> for the query options you can adjust using this + statement. + </p> + </div> +<nav role="navigation" class="related-links"><ul class="ullinks"><li class="link ulchildlink"><strong><a href="../topics/impala_query_options.html">Query Options for the SET Statement</a></strong><br></li></ul><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_shell_commands.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_shell_commands.html b/docs/build/html/topics/impala_shell_commands.html new file mode 100644 index 0000000..d2bee6c --- /dev/null +++ b/docs/build/html/topics/impala_shell_commands.html @@ -0,0 +1,392 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_impala_shell.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="shell_commands"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>impala-shell Command Reference</title></head><body id="shell_commands"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">impala-shell Command Reference</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Use the following commands within <code class="ph codeph">impala-shell</code> to pass requests to the + <code class="ph codeph">impalad</code> daemon that the shell is connected to. You can enter a command interactively at the + prompt, or pass it as the argument to the <code class="ph codeph">-q</code> option of <code class="ph codeph">impala-shell</code>. Most + of these commands are passed to the Impala daemon as SQL statements; refer to the corresponding + <a class="xref" href="impala_langref_sql.html#langref_sql">SQL language reference sections</a> for full syntax + details. + </p> + + <table class="table"><caption></caption><colgroup><col style="width:20%"><col style="width:80%"></colgroup><thead class="thead"> + <tr class="row"> + <th class="entry nocellnorowborder" id="shell_commands__entry__1"> + Command + </th> + <th class="entry nocellnorowborder" id="shell_commands__entry__2"> + Explanation + </th> + </tr> + </thead><tbody class="tbody"> + <tr class="row" id="shell_commands__alter_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">alter</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Changes the underlying structure or settings of an Impala table, or a table shared between Impala + and Hive. See <a class="xref" href="impala_alter_table.html#alter_table">ALTER TABLE Statement</a> and + <a class="xref" href="impala_alter_view.html#alter_view">ALTER VIEW Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__compute_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">compute stats</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Gathers important performance-related information for a table, used by Impala to optimize queries. + See <a class="xref" href="impala_compute_stats.html#compute_stats">COMPUTE STATS Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__connect_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">connect</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Connects to the specified instance of <code class="ph codeph">impalad</code>. The default port of 21000 is + assumed unless you provide another value. You can connect to any host in your cluster that is + running <code class="ph codeph">impalad</code>. If you connect to an instance of <code class="ph codeph">impalad</code> that + was started with an alternate port specified by the <code class="ph codeph">--fe_port</code> flag, you must + provide that alternate port. See <a class="xref" href="impala_connecting.html#connecting">Connecting to impalad through impala-shell</a> for examples. + </p> + + <p class="p"> + The <code class="ph codeph">SET</code> statement has no effect until the <span class="keyword cmdname">impala-shell</span> interpreter is + connected to an Impala server. Once you are connected, any query options you set remain in effect as you + issue a subsequent <code class="ph codeph">CONNECT</code> command to connect to a different Impala host. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__describe_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">describe</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Shows the columns, column data types, and any column comments for a specified table. + <code class="ph codeph">DESCRIBE FORMATTED</code> shows additional information such as the HDFS data directory, + partitions, and internal properties for the table. See <a class="xref" href="impala_describe.html#describe">DESCRIBE Statement</a> + for details about the basic <code class="ph codeph">DESCRIBE</code> output and the <code class="ph codeph">DESCRIBE + FORMATTED</code> variant. You can use <code class="ph codeph">DESC</code> as shorthand for the + <code class="ph codeph">DESCRIBE</code> command. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__drop_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">drop</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Removes a schema object, and in some cases its associated data files. See + <a class="xref" href="impala_drop_table.html#drop_table">DROP TABLE Statement</a>, <a class="xref" href="impala_drop_view.html#drop_view">DROP VIEW Statement</a>, + <a class="xref" href="impala_drop_database.html#drop_database">DROP DATABASE Statement</a>, and + <a class="xref" href="impala_drop_function.html#drop_function">DROP FUNCTION Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__explain_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">explain</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Provides the execution plan for a query. <code class="ph codeph">EXPLAIN</code> represents a query as a series of + steps. For example, these steps might be map/reduce stages, metastore operations, or file system + operations such as move or rename. See <a class="xref" href="impala_explain.html#explain">EXPLAIN Statement</a> and + <a class="xref" href="impala_explain_plan.html#perf_explain">Using the EXPLAIN Plan for Performance Tuning</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__help_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">help</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Help provides a list of all available commands and options. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__history_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">history</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Maintains an enumerated cross-session command history. This history is stored in the + <span class="ph filepath">~/.impalahistory</span> file. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__insert_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">insert</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Writes the results of a query to a specified table. This either overwrites table data or appends + data to the existing table content. See <a class="xref" href="impala_insert.html#insert">INSERT Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__invalidate_metadata_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">invalidate metadata</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Updates <span class="keyword cmdname">impalad</span> metadata for table existence and structure. Use this command + after creating, dropping, or altering databases, tables, or partitions in Hive. See + <a class="xref" href="impala_invalidate_metadata.html#invalidate_metadata">INVALIDATE METADATA Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__profile_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">profile</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Displays low-level information about the most recent query. Used for performance diagnosis and + tuning. <span class="ph"> The report starts with the same information as produced by the + <code class="ph codeph">EXPLAIN</code> statement and the <code class="ph codeph">SUMMARY</code> command.</span> See + <a class="xref" href="impala_explain_plan.html#perf_profile">Using the Query Profile for Performance Tuning</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__quit_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">quit</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Exits the shell. Remember to include the final semicolon so that the shell recognizes the end of + the command. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__refresh_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">refresh</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Refreshes <span class="keyword cmdname">impalad</span> metadata for the locations of HDFS blocks corresponding to + Impala data files. Use this command after loading new data files into an Impala table through Hive + or through HDFS commands. See <a class="xref" href="impala_refresh.html#refresh">REFRESH Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__select_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">select</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Specifies the data set on which to complete some action. All information returned from + <code class="ph codeph">select</code> can be sent to some output such as the console or a file or can be used to + complete some other element of query. See <a class="xref" href="impala_select.html#select">SELECT Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__set_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">set</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Manages query options for an <span class="keyword cmdname">impala-shell</span> session. The available options are the + ones listed in <a class="xref" href="impala_query_options.html#query_options">Query Options for the SET Statement</a>. These options are used for + query tuning and troubleshooting. Issue <code class="ph codeph">SET</code> with no arguments to see the current + query options, either based on the <span class="keyword cmdname">impalad</span> defaults, as specified by you at + <span class="keyword cmdname">impalad</span> startup, or based on earlier <code class="ph codeph">SET</code> statements in the same + session. To modify option values, issue commands with the syntax <code class="ph codeph">set + <var class="keyword varname">option</var>=<var class="keyword varname">value</var></code>. To restore an option to its default, + use the <code class="ph codeph">unset</code> command. Some options take Boolean values of <code class="ph codeph">true</code> + and <code class="ph codeph">false</code>. Others take numeric arguments, or quoted string values. + </p> + + <p class="p"> + The <code class="ph codeph">SET</code> statement has no effect until the <span class="keyword cmdname">impala-shell</span> interpreter is + connected to an Impala server. Once you are connected, any query options you set remain in effect as you + issue a subsequent <code class="ph codeph">CONNECT</code> command to connect to a different Impala host. + </p> + + <p class="p"> + In Impala 2.0 and later, <code class="ph codeph">SET</code> is available as a SQL statement for any kind of + application, not only through <span class="keyword cmdname">impala-shell</span>. See + <a class="xref" href="impala_set.html#set">SET Statement</a> for details. + </p> + + <p class="p"> + In Impala 2.5 and later, you can use <code class="ph codeph">SET</code> to define your own substitution variables + within an <span class="keyword cmdname">impala-shell</span> session. + Within a SQL statement, you substitute the value by using the notation <code class="ph codeph">${var:<var class="keyword varname">variable_name</var>}</code>. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__shell_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">shell</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Executes the specified command in the operating system shell without exiting + <code class="ph codeph">impala-shell</code>. You can use the <code class="ph codeph">!</code> character as shorthand for the + <code class="ph codeph">shell</code> command. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + Quote any instances of the <code class="ph codeph">--</code> or <code class="ph codeph">/*</code> tokens to avoid them being + interpreted as the start of a comment. To embed comments within <code class="ph codeph">source</code> or + <code class="ph codeph">!</code> commands, use the shell comment character <code class="ph codeph">#</code> before the comment + portion of the line. + </div> + </td> + </tr> + <tr class="row" id="shell_commands__show_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">show</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Displays metastore data for schema objects created and accessed through Impala, Hive, or both. + <code class="ph codeph">show</code> can be used to gather information about objects such as databases, tables, and functions. + See <a class="xref" href="impala_show.html#show">SHOW Statement</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__source_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">source</code> or <code class="ph codeph">src</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Executes one or more statements residing in a specified file from the local filesystem. + Allows you to perform the same kinds of batch operations as with the <code class="ph codeph">-f</code> option, + but interactively within the interpreter. The file can contain SQL statements and other + <span class="keyword cmdname">impala-shell</span> commands, including additional <code class="ph codeph">SOURCE</code> commands + to perform a flexible sequence of actions. Each command or statement, except the last one in the file, + must end with a semicolon. + See <a class="xref" href="impala_shell_running_commands.html#shell_running_commands">Running Commands and SQL Statements in impala-shell</a> for examples. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__summary_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">summary</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Summarizes the work performed in various stages of a query. It provides a higher-level view of the + information displayed by the <code class="ph codeph">EXPLAIN</code> command. Added in Impala 1.4.0. See + <a class="xref" href="impala_explain_plan.html#perf_summary">Using the SUMMARY Report for Performance Tuning</a> for details about the report format + and how to interpret it. + </p> + <p class="p"> + In <span class="keyword">Impala 2.3</span> and higher, you can see a continuously updated report of + the summary information while a query is in progress. + See <a class="xref" href="impala_live_summary.html#live_summary">LIVE_SUMMARY Query Option (Impala 2.3 or higher only)</a> for details. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__unset_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">unset</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Removes any user-specified value for a query option and returns the option to its default value. + See <a class="xref" href="impala_query_options.html#query_options">Query Options for the SET Statement</a> for the available query options. + </p> + <p class="p"> + In <span class="keyword">Impala 2.5</span> and higher, it can also remove user-specified substitution variables + using the notation <code class="ph codeph">UNSET VAR:<var class="keyword varname">variable_name</var></code>. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__use_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">use</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Indicates the database against which to execute subsequent commands. Lets you avoid using fully + qualified names when referring to tables in databases other than <code class="ph codeph">default</code>. See + <a class="xref" href="impala_use.html#use">USE Statement</a> for details. Not effective with the <code class="ph codeph">-q</code> option, + because that option only allows a single statement in the argument. + </p> + </td> + </tr> + <tr class="row" id="shell_commands__version_cmd"> + <td class="entry nocellnorowborder" headers="shell_commands__entry__1 "> + <p class="p"> + <code class="ph codeph">version</code> + </p> + </td> + <td class="entry nocellnorowborder" headers="shell_commands__entry__2 "> + <p class="p"> + Returns Impala version information. + </p> + </td> + </tr> + </tbody></table> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_impala_shell.html">Using the Impala Shell (impala-shell Command)</a></div></div></nav></article></main></body></html> \ No newline at end of file
