http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_distinct.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_distinct.html b/docs/build/html/topics/impala_distinct.html new file mode 100644 index 0000000..dbdac24 --- /dev/null +++ b/docs/build/html/topics/impala_distinct.html @@ -0,0 +1,81 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_select.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="distinct"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DISTINCT Operator</title></head><body id="distinct"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DISTINCT Operator</h1> + + + <div class="body conbody"> + + <p class="p"> + + The <code class="ph codeph">DISTINCT</code> operator in a <code class="ph codeph">SELECT</code> statement filters the result set to + remove duplicates: + </p> + +<pre class="pre codeblock"><code>-- Returns the unique values from one column. +-- NULL is included in the set of values if any rows have a NULL in this column. +select distinct c_birth_country from customer; +-- Returns the unique combinations of values from multiple columns. +select distinct c_salutation, c_last_name from customer;</code></pre> + + <p class="p"> + You can use <code class="ph codeph">DISTINCT</code> in combination with an aggregation function, typically + <code class="ph codeph">COUNT()</code>, to find how many different values a column contains: + </p> + +<pre class="pre codeblock"><code>-- Counts the unique values from one column. +-- NULL is not included as a distinct value in the count. +select count(distinct c_birth_country) from customer; +-- Counts the unique combinations of values from multiple columns. +select count(distinct c_salutation, c_last_name) from customer;</code></pre> + + <p class="p"> + One construct that Impala SQL does <em class="ph i">not</em> support is using <code class="ph codeph">DISTINCT</code> in more than one + aggregation function in the same query. For example, you could not have a single query with both + <code class="ph codeph">COUNT(DISTINCT c_first_name)</code> and <code class="ph codeph">COUNT(DISTINCT c_last_name)</code> in the + <code class="ph codeph">SELECT</code> list. + </p> + + <p class="p"> + <strong class="ph b">Zero-length strings:</strong> For purposes of clauses such as <code class="ph codeph">DISTINCT</code> and <code class="ph codeph">GROUP + BY</code>, Impala considers zero-length strings (<code class="ph codeph">""</code>), <code class="ph codeph">NULL</code>, and space + to all be different values. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + By default, Impala only allows a single <code class="ph codeph">COUNT(DISTINCT <var class="keyword varname">columns</var>)</code> + expression in each query. + </p> + <p class="p"> + If you do not need precise accuracy, you can produce an estimate of the distinct values for a column by + specifying <code class="ph codeph">NDV(<var class="keyword varname">column</var>)</code>; a query can contain multiple instances of + <code class="ph codeph">NDV(<var class="keyword varname">column</var>)</code>. To make Impala automatically rewrite + <code class="ph codeph">COUNT(DISTINCT)</code> expressions to <code class="ph codeph">NDV()</code>, enable the + <code class="ph codeph">APPX_COUNT_DISTINCT</code> query option. + </p> + <p class="p"> + To produce the same result as multiple <code class="ph codeph">COUNT(DISTINCT)</code> expressions, you can use the + following technique for queries involving a single table: + </p> +<pre class="pre codeblock"><code>select v1.c1 result1, v2.c1 result2 from + (select count(distinct col1) as c1 from t1) v1 + cross join + (select count(distinct col2) as c1 from t1) v2; +</code></pre> + <p class="p"> + Because <code class="ph codeph">CROSS JOIN</code> is an expensive operation, prefer to use the <code class="ph codeph">NDV()</code> + technique wherever practical. + </p> + </div> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + In contrast with some database systems that always return <code class="ph codeph">DISTINCT</code> values in sorted order, + Impala does not do any ordering of <code class="ph codeph">DISTINCT</code> values. Always include an <code class="ph codeph">ORDER + BY</code> clause if you need the values in alphabetical or numeric sorted order. + </p> + </div> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_select.html">SELECT Statement</a></div></div></nav></article></main></body></html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_dml.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_dml.html b/docs/build/html/topics/impala_dml.html new file mode 100644 index 0000000..71b7158 --- /dev/null +++ b/docs/build/html/topics/impala_dml.html @@ -0,0 +1,82 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="dml"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DML Statements</title></head><body id="dml"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DML Statements</h1> + + + <div class="body conbody"> + + <p class="p"> + DML refers to <span class="q">"Data Manipulation Language"</span>, a subset of SQL statements that modify the data stored in + tables. Because Impala focuses on query performance and leverages the append-only nature of HDFS storage, + currently Impala only supports a small set of DML statements: + </p> + + <ul class="ul"> + <li class="li"> + <a class="xref" href="impala_delete.html">DELETE Statement (Impala 2.8 or higher only)</a>. Works for Kudu tables only. + </li> + + <li class="li"> + <a class="xref" href="impala_insert.html">INSERT Statement</a>. + </li> + + <li class="li"> + <a class="xref" href="impala_load_data.html">LOAD DATA Statement</a>. Does not apply for HBase or Kudu tables. + </li> + + <li class="li"> + <a class="xref" href="impala_update.html">UPDATE Statement (Impala 2.8 or higher only)</a>. Works for Kudu tables only. + </li> + + <li class="li"> + <a class="xref" href="impala_upsert.html">UPSERT Statement (Impala 2.8 or higher only)</a>. Works for Kudu tables only. + </li> + </ul> + + <p class="p"> + <code class="ph codeph">INSERT</code> in Impala is primarily optimized for inserting large volumes of data in a single + statement, to make effective use of the multi-megabyte HDFS blocks. This is the way in Impala to create new + data files. If you intend to insert one or a few rows at a time, such as using the <code class="ph codeph">INSERT ... + VALUES</code> syntax, that technique is much more efficient for Impala tables stored in HBase. See + <a class="xref" href="impala_hbase.html#impala_hbase">Using Impala to Query HBase Tables</a> for details. + </p> + + <p class="p"> + <code class="ph codeph">LOAD DATA</code> moves existing data files into the directory for an Impala table, making them + immediately available for Impala queries. This is one way in Impala to work with data files produced by other + Hadoop components. (<code class="ph codeph">CREATE EXTERNAL TABLE</code> is the other alternative; with external tables, + you can query existing data files, while the files remain in their original location.) + </p> + + <p class="p"> + In <span class="keyword">Impala 2.8</span> and higher, Impala does support the <code class="ph codeph">UPDATE</code>, <code class="ph codeph">DELETE</code>, + and <code class="ph codeph">UPSERT</code> statements for Kudu tables. + For HDFS or S3 tables, to simulate the effects of an <code class="ph codeph">UPDATE</code> or <code class="ph codeph">DELETE</code> statement + in other database systems, typically you use <code class="ph codeph">INSERT</code> or <code class="ph codeph">CREATE TABLE AS SELECT</code> to copy data + from one table to another, filtering out or changing the appropriate rows during the copy operation. + </p> + + <p class="p"> + You can also achieve a result similar to <code class="ph codeph">UPDATE</code> by using Impala tables stored in HBase. + When you insert a row into an HBase table, and the table + already contains a row with the same value for the key column, the older row is hidden, effectively the same + as a single-row <code class="ph codeph">UPDATE</code>. + </p> + + <p class="p"> + Impala can perform DML operations for tables or partitions stored in the Amazon S3 filesystem + with <span class="keyword">Impala 2.6</span> and higher. See <a class="xref" href="impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + The other major classifications of SQL statements are data definition language (see + <a class="xref" href="impala_ddl.html#ddl">DDL Statements</a>) and queries (see <a class="xref" href="impala_select.html#select">SELECT Statement</a>). + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_double.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_double.html b/docs/build/html/topics/impala_double.html new file mode 100644 index 0000000..b87994c --- /dev/null +++ b/docs/build/html/topics/impala_double.html @@ -0,0 +1,144 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_datatypes.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="double"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DOUBLE Data Type</title></head><body id="double"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DOUBLE Data Type</h1> + + + + <div class="body conbody"> + + <p class="p"> + A double precision floating-point data type used in <code class="ph codeph">CREATE TABLE</code> and <code class="ph codeph">ALTER + TABLE</code> statements. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + + <p class="p"> + In the column definition of a <code class="ph codeph">CREATE TABLE</code> statement: + </p> + +<pre class="pre codeblock"><code><var class="keyword varname">column_name</var> DOUBLE</code></pre> + + <p class="p"> + <strong class="ph b">Range:</strong> 4.94065645841246544e-324d .. 1.79769313486231570e+308, positive or negative + </p> + + <p class="p"> + <strong class="ph b">Precision:</strong> 15 to 17 significant digits, depending on usage. The number of significant digits does + not depend on the position of the decimal point. + </p> + + <p class="p"> + <strong class="ph b">Representation:</strong> The values are stored in 8 bytes, using + <a class="xref" href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format"; target="_blank">IEEE 754 Double Precision Binary Floating Point</a> format. + </p> + + <p class="p"> + <strong class="ph b">Conversions:</strong> Impala does not automatically convert <code class="ph codeph">DOUBLE</code> to any other type. You can + use <code class="ph codeph">CAST()</code> to convert <code class="ph codeph">DOUBLE</code> values to <code class="ph codeph">FLOAT</code>, + <code class="ph codeph">TINYINT</code>, <code class="ph codeph">SMALLINT</code>, <code class="ph codeph">INT</code>, <code class="ph codeph">BIGINT</code>, + <code class="ph codeph">STRING</code>, <code class="ph codeph">TIMESTAMP</code>, or <code class="ph codeph">BOOLEAN</code>. You can use exponential + notation in <code class="ph codeph">DOUBLE</code> literals or when casting from <code class="ph codeph">STRING</code>, for example + <code class="ph codeph">1.0e6</code> to represent one million. + <span class="ph">Casting an integer or floating-point value <code class="ph codeph">N</code> to + <code class="ph codeph">TIMESTAMP</code> produces a value that is <code class="ph codeph">N</code> seconds past the start of the epoch + date (January 1, 1970). By default, the result value represents a date and time in the UTC time zone. + If the setting <code class="ph codeph">-use_local_tz_for_unix_timestamp_conversions=true</code> is in effect, + the resulting <code class="ph codeph">TIMESTAMP</code> represents a date and time in the local time zone.</span> + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + The data type <code class="ph codeph">REAL</code> is an alias for <code class="ph codeph">DOUBLE</code>. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + +<pre class="pre codeblock"><code>CREATE TABLE t1 (x DOUBLE); +SELECT CAST(1000.5 AS DOUBLE); +</code></pre> + + <p class="p"> + <strong class="ph b">Partitioning:</strong> Because fractional values of this type are not always represented precisely, when this + type is used for a partition key column, the underlying HDFS directories might not be named exactly as you + expect. Prefer to partition on a <code class="ph codeph">DECIMAL</code> column instead. + </p> + + <p class="p"> + <strong class="ph b">HBase considerations:</strong> This data type is fully compatible with HBase tables. + </p> + + <p class="p"> + <strong class="ph b">Parquet considerations:</strong> This type is fully compatible with Parquet tables. + </p> + + <p class="p"> + <strong class="ph b">Text table considerations:</strong> Values of this type are potentially larger in text tables than in tables + using Parquet or other binary formats. + </p> + + + + <p class="p"> + <strong class="ph b">Internal details:</strong> Represented in memory as an 8-byte value. + </p> + + + + <p class="p"> + <strong class="ph b">Column statistics considerations:</strong> Because this type has a fixed size, the maximum and average size + fields are always filled in for column statistics, even before you run the <code class="ph codeph">COMPUTE STATS</code> + statement. + </p> + + <p class="p"> + <strong class="ph b">Restrictions:</strong> + </p> + + + + <p class="p"> + Due to the way arithmetic on <code class="ph codeph">FLOAT</code> and <code class="ph codeph">DOUBLE</code> columns uses + high-performance hardware instructions, and distributed queries can perform these operations in different + order for each query, results can vary slightly for aggregate function calls such as <code class="ph codeph">SUM()</code> + and <code class="ph codeph">AVG()</code> for <code class="ph codeph">FLOAT</code> and <code class="ph codeph">DOUBLE</code> columns, particularly on + large data sets where millions or billions of values are summed or averaged. For perfect consistency and + repeatability, use the <code class="ph codeph">DECIMAL</code> data type for such operations instead of + <code class="ph codeph">FLOAT</code> or <code class="ph codeph">DOUBLE</code>. + </p> + + <p class="p"> + The inability to exactly represent certain floating-point values means that + <code class="ph codeph">DECIMAL</code> is sometimes a better choice than <code class="ph codeph">DOUBLE</code> + or <code class="ph codeph">FLOAT</code> when precision is critical, particularly when + transferring data from other database systems that use different representations + or file formats. + </p> + + <p class="p"> + <strong class="ph b">Kudu considerations:</strong> + </p> + <p class="p"> + Currently, the data types <code class="ph codeph">BOOLEAN</code>, <code class="ph codeph">FLOAT</code>, + and <code class="ph codeph">DOUBLE</code> cannot be used for primary key columns in Kudu tables. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_literals.html#numeric_literals">Numeric Literals</a>, <a class="xref" href="impala_math_functions.html#math_functions">Impala Mathematical Functions</a>, + <a class="xref" href="impala_float.html#float">FLOAT Data Type</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_datatypes.html">Data Types</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_database.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_database.html b/docs/build/html/topics/impala_drop_database.html new file mode 100644 index 0000000..1e974df --- /dev/null +++ b/docs/build/html/topics/impala_drop_database.html @@ -0,0 +1,193 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_database"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP DATABASE Statement</title></head><body id="drop_database"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP DATABASE Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Removes a database from the system. The physical operations involve removing the metadata for the database + from the metastore, and deleting the corresponding <code class="ph codeph">*.db</code> directory from HDFS. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>DROP (DATABASE|SCHEMA) [IF EXISTS] <var class="keyword varname">database_name</var> <span class="ph">[RESTRICT | CASCADE]</span>;</code></pre> + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + By default, the database must be empty before it can be dropped, to avoid losing any data. + </p> + + <p class="p"> + In <span class="keyword">Impala 2.3</span> and higher, you can include the <code class="ph codeph">CASCADE</code> + clause to make Impala drop all tables and other objects in the database before dropping the database itself. + The <code class="ph codeph">RESTRICT</code> clause enforces the original requirement that the database be empty + before being dropped. Because the <code class="ph codeph">RESTRICT</code> behavior is still the default, this + clause is optional. + </p> + + <p class="p"> + The automatic dropping resulting from the <code class="ph codeph">CASCADE</code> clause follows the same rules as the + corresponding <code class="ph codeph">DROP TABLE</code>, <code class="ph codeph">DROP VIEW</code>, and <code class="ph codeph">DROP FUNCTION</code> statements. + In particular, the HDFS directories and data files for any external tables are left behind when the + tables are removed. + </p> + + <p class="p"> + When you do not use the <code class="ph codeph">CASCADE</code> clause, drop or move all the objects inside the database manually + before dropping the database itself: + </p> + + <ul class="ul"> + <li class="li"> + <p class="p"> + Use the <code class="ph codeph">SHOW TABLES</code> statement to locate all tables and views in the database, + and issue <code class="ph codeph">DROP TABLE</code> and <code class="ph codeph">DROP VIEW</code> statements to remove them all. + </p> + </li> + <li class="li"> + <p class="p"> + Use the <code class="ph codeph">SHOW FUNCTIONS</code> and <code class="ph codeph">SHOW AGGREGATE FUNCTIONS</code> statements + to locate all user-defined functions in the database, and issue <code class="ph codeph">DROP FUNCTION</code> + and <code class="ph codeph">DROP AGGREGATE FUNCTION</code> statements to remove them all. + </p> + </li> + <li class="li"> + <p class="p"> + To keep tables or views contained by a database while removing the database itself, use + <code class="ph codeph">ALTER TABLE</code> and <code class="ph codeph">ALTER VIEW</code> to move the relevant + objects to a different database before dropping the original database. + </p> + </li> + </ul> + + <p class="p"> + You cannot drop the current database, that is, the database your session connected to + either through the <code class="ph codeph">USE</code> statement or the <code class="ph codeph">-d</code> option of <span class="keyword cmdname">impala-shell</span>. + Issue a <code class="ph codeph">USE</code> statement to switch to a different database first. + Because the <code class="ph codeph">default</code> database is always available, issuing + <code class="ph codeph">USE default</code> is a convenient way to leave the current database + before dropping it. + </p> + + <p class="p"> + <strong class="ph b">Hive considerations:</strong> + </p> + + <p class="p"> + When you drop a database in Impala, the database can no longer be used by Hive. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + + + <p class="p"> + See <a class="xref" href="impala_create_database.html#create_database">CREATE DATABASE Statement</a> for examples covering <code class="ph codeph">CREATE + DATABASE</code>, <code class="ph codeph">USE</code>, and <code class="ph codeph">DROP DATABASE</code>. + </p> + + <p class="p"> + <strong class="ph b">Amazon S3 considerations:</strong> + </p> + + <p class="p"> + In <span class="keyword">Impala 2.6</span> and higher, Impala DDL statements such as + <code class="ph codeph">CREATE DATABASE</code>, <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">DROP DATABASE CASCADE</code>, + <code class="ph codeph">DROP TABLE</code>, and <code class="ph codeph">ALTER TABLE [ADD|DROP] PARTITION</code> can create or remove folders + as needed in the Amazon S3 system. Prior to <span class="keyword">Impala 2.6</span>, you had to create folders yourself and point + Impala database, tables, or partitions at them, and manually remove folders when no longer needed. + See <a class="xref" href="../shared/../topics/impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about reading and writing S3 data with Impala. + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, must have write + permission for the directory associated with the database. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <pre class="pre codeblock"><code>create database first_db; +use first_db; +create table t1 (x int); + +create database second_db; +use second_db; +-- Each database has its own namespace for tables. +-- You can reuse the same table names in each database. +create table t1 (s string); + +create database temp; + +-- You can either USE a database after creating it, +-- or qualify all references to the table name with the name of the database. +-- Here, tables T2 and T3 are both created in the TEMP database. + +create table temp.t2 (x int, y int); +use database temp; +create table t3 (s string); + +-- You cannot drop a database while it is selected by the USE statement. +drop database temp; +<em class="ph i">ERROR: AnalysisException: Cannot drop current default database: temp</em> + +-- The always-available database 'default' is a convenient one to USE +-- before dropping a database you created. +use default; + +-- Before dropping a database, first drop all the tables inside it, +<span class="ph">-- or in <span class="keyword">Impala 2.3</span> and higher use the CASCADE clause.</span> +drop database temp; +ERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: +CAUSED BY: InvalidOperationException: Database temp is not empty +show tables in temp; ++------+ +| name | ++------+ +| t3 | ++------+ + +<span class="ph">-- <span class="keyword">Impala 2.3</span> and higher:</span> +<span class="ph">drop database temp cascade;</span> + +-- Earlier releases: +drop table temp.t3; +drop database temp; +</code></pre> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_databases.html#databases">Overview of Impala Databases</a>, <a class="xref" href="impala_create_database.html#create_database">CREATE DATABASE Statement</a>, + <a class="xref" href="impala_use.html#use">USE Statement</a>, <a class="xref" href="impala_show.html#show_databases">SHOW DATABASES</a>, <a class="xref" href="impala_drop_table.html#drop_table">DROP TABLE Statement</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_function.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_function.html b/docs/build/html/topics/impala_drop_function.html new file mode 100644 index 0000000..fd9f839 --- /dev/null +++ b/docs/build/html/topics/impala_drop_function.html @@ -0,0 +1,136 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_function"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP FUNCTION Statement</title></head><body id="drop_function"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP FUNCTION Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Removes a user-defined function (UDF), so that it is not available for execution during Impala + <code class="ph codeph">SELECT</code> or <code class="ph codeph">INSERT</code> operations. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + + <p class="p"> + To drop C++ UDFs and UDAs: + </p> + +<pre class="pre codeblock"><code>DROP [AGGREGATE] FUNCTION [IF EXISTS] [<var class="keyword varname">db_name</var>.]<var class="keyword varname">function_name</var>(<var class="keyword varname">type</var>[, <var class="keyword varname">type</var>...])</code></pre> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + <p class="p"> + The preceding syntax, which includes the function signature, also applies to Java UDFs that were created + using the corresponding <code class="ph codeph">CREATE FUNCTION</code> syntax that includes the argument and return types. + After upgrading to <span class="keyword">Impala 2.5</span> or higher, consider re-creating all Java UDFs with the + <code class="ph codeph">CREATE FUNCTION</code> syntax that does not include the function signature. Java UDFs created this + way are now persisted in the metastore database and do not need to be re-created after an Impala restart. + </p> + </div> + + <p class="p"> + To drop Java UDFs (created using the <code class="ph codeph">CREATE FUNCTION</code> syntax with no function signature): + </p> + +<pre class="pre codeblock"><code>DROP FUNCTION [IF EXISTS] [<var class="keyword varname">db_name</var>.]<var class="keyword varname">function_name</var></code></pre> + + + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + Because the same function name could be overloaded with different argument signatures, you specify the + argument types to identify the exact function to drop. + </p> + + <p class="p"> + <strong class="ph b">Restrictions:</strong> + </p> + + <p class="p"> + In <span class="keyword">Impala 2.5</span> and higher, Impala UDFs and UDAs written in C++ are persisted in the metastore database. + Java UDFs are also persisted, if they were created with the new <code class="ph codeph">CREATE FUNCTION</code> syntax for Java UDFs, + where the Java function argument and return types are omitted. + Java-based UDFs created with the old <code class="ph codeph">CREATE FUNCTION</code> syntax do not persist across restarts + because they are held in the memory of the <span class="keyword cmdname">catalogd</span> daemon. + Until you re-create such Java UDFs using the new <code class="ph codeph">CREATE FUNCTION</code> syntax, + you must reload those Java-based UDFs by running the original <code class="ph codeph">CREATE FUNCTION</code> statements again each time + you restart the <span class="keyword cmdname">catalogd</span> daemon. + Prior to <span class="keyword">Impala 2.5</span> the requirement to reload functions after a restart applied to both C++ and Java functions. + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, does not need any + particular HDFS permissions to perform this statement. + All read and write operations are on the metastore database, + not HDFS files and directories. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + <p class="p"> + The following example shows how to drop Java functions created with the signatureless + <code class="ph codeph">CREATE FUNCTION</code> syntax in <span class="keyword">Impala 2.5</span> and higher. + Issuing <code class="ph codeph">DROP FUNCTION <var class="keyword varname">function_name</var></code> removes all the + overloaded functions under that name. + (See <a class="xref" href="impala_create_function.html#create_function">CREATE FUNCTION Statement</a> for a longer example + showing how to set up such functions in the first place.) + </p> +<pre class="pre codeblock"><code> +create function my_func location '/user/impala/udfs/udf-examples.jar' + symbol='org.apache.impala.TestUdf'; + +show functions; ++-------------+---------------------------------------+-------------+---------------+ +| return type | signature | binary type | is persistent | ++-------------+---------------------------------------+-------------+---------------+ +| BIGINT | my_func(BIGINT) | JAVA | true | +| BOOLEAN | my_func(BOOLEAN) | JAVA | true | +| BOOLEAN | my_func(BOOLEAN, BOOLEAN) | JAVA | true | +... +| BIGINT | testudf(BIGINT) | JAVA | true | +| BOOLEAN | testudf(BOOLEAN) | JAVA | true | +| BOOLEAN | testudf(BOOLEAN, BOOLEAN) | JAVA | true | +... + +drop function my_func; +show functions; ++-------------+---------------------------------------+-------------+---------------+ +| return type | signature | binary type | is persistent | ++-------------+---------------------------------------+-------------+---------------+ +| BIGINT | testudf(BIGINT) | JAVA | true | +| BOOLEAN | testudf(BOOLEAN) | JAVA | true | +| BOOLEAN | testudf(BOOLEAN, BOOLEAN) | JAVA | true | +... +</code></pre> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_udf.html#udfs">Impala User-Defined Functions (UDFs)</a>, <a class="xref" href="impala_create_function.html#create_function">CREATE FUNCTION Statement</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_role.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_role.html b/docs/build/html/topics/impala_drop_role.html new file mode 100644 index 0000000..addaf76 --- /dev/null +++ b/docs/build/html/topics/impala_drop_role.html @@ -0,0 +1,71 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_role"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP ROLE Statement (Impala 2.0 or higher only)</title></head><body id="drop_role"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP ROLE Statement (<span class="keyword">Impala 2.0</span> or higher only)</h1> + + + + <div class="body conbody"> + + <p class="p"> + + + The <code class="ph codeph">DROP ROLE</code> statement removes a role from the metastore database. Once dropped, the role + is revoked for all users to whom it was previously assigned, and all privileges granted to that role are + revoked. Queries that are already executing are not affected. Impala verifies the role information + approximately every 60 seconds, so the effects of <code class="ph codeph">DROP ROLE</code> might not take effect for new + Impala queries for a brief period. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>DROP ROLE <var class="keyword varname">role_name</var> +</code></pre> + + <p class="p"> + <strong class="ph b">Required privileges:</strong> + </p> + + <p class="p"> + Only administrative users (initially, a predefined set of users specified in the Sentry service configuration + file) can use this statement. + </p> + + <p class="p"> + <strong class="ph b">Compatibility:</strong> + </p> + + <p class="p"> + Impala makes use of any roles and privileges specified by the <code class="ph codeph">GRANT</code> and + <code class="ph codeph">REVOKE</code> statements in Hive, and Hive makes use of any roles and privileges specified by the + <code class="ph codeph">GRANT</code> and <code class="ph codeph">REVOKE</code> statements in Impala. The Impala <code class="ph codeph">GRANT</code> + and <code class="ph codeph">REVOKE</code> statements for privileges do not require the <code class="ph codeph">ROLE</code> keyword to be + repeated before each role name, unlike the equivalent Hive statements. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_authorization.html#authorization">Enabling Sentry Authorization for Impala</a>, <a class="xref" href="impala_grant.html#grant">GRANT Statement (Impala 2.0 or higher only)</a> + <a class="xref" href="impala_revoke.html#revoke">REVOKE Statement (Impala 2.0 or higher only)</a>, <a class="xref" href="impala_create_role.html#create_role">CREATE ROLE Statement (Impala 2.0 or higher only)</a>, + <a class="xref" href="impala_show.html#show">SHOW Statement</a> + </p> + + + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> This statement does not touch any HDFS files or directories, + therefore no HDFS permissions are required. + </p> + + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_stats.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_stats.html b/docs/build/html/topics/impala_drop_stats.html new file mode 100644 index 0000000..f023867 --- /dev/null +++ b/docs/build/html/topics/impala_drop_stats.html @@ -0,0 +1,285 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_stats"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP STATS Statement</title></head><body id="drop_stats"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP STATS Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Removes the specified statistics from a table or partition. The statistics were originally created by the + <code class="ph codeph">COMPUTE STATS</code> or <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>DROP STATS [<var class="keyword varname">database_name</var>.]<var class="keyword varname">table_name</var> +DROP INCREMENTAL STATS [<var class="keyword varname">database_name</var>.]<var class="keyword varname">table_name</var> PARTITION (<var class="keyword varname">partition_spec</var>) + +<var class="keyword varname">partition_spec</var> ::= <var class="keyword varname">partition_col</var>=<var class="keyword varname">constant_value</var> +</code></pre> + + <p class="p"> + The <code class="ph codeph">PARTITION</code> clause is only allowed in combination with the <code class="ph codeph">INCREMENTAL</code> + clause. It is optional for <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>, and required for <code class="ph codeph">DROP + INCREMENTAL STATS</code>. Whenever you specify partitions through the <code class="ph codeph">PARTITION + (<var class="keyword varname">partition_spec</var>)</code> clause in a <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> or + <code class="ph codeph">DROP INCREMENTAL STATS</code> statement, you must include all the partitioning columns in the + specification, and specify constant values for all the partition key columns. + </p> + + <p class="p"> + <code class="ph codeph">DROP STATS</code> removes all statistics from the table, whether created by <code class="ph codeph">COMPUTE + STATS</code> or <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>. + </p> + + <p class="p"> + <code class="ph codeph">DROP INCREMENTAL STATS</code> only affects incremental statistics for a single partition, specified + through the <code class="ph codeph">PARTITION</code> clause. The incremental stats are marked as outdated, so that they are + recomputed by the next <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement. + </p> + + + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + You typically use this statement when the statistics for a table or a partition have become stale due to data + files being added to or removed from the associated HDFS data directories, whether by manual HDFS operations + or <code class="ph codeph">INSERT</code>, <code class="ph codeph">INSERT OVERWRITE</code>, or <code class="ph codeph">LOAD DATA</code> statements, or + adding or dropping partitions. + </p> + + <p class="p"> + When a table or partition has no associated statistics, Impala treats it as essentially zero-sized when + constructing the execution plan for a query. In particular, the statistics influence the order in which + tables are joined in a join query. To ensure proper query planning and good query performance and + scalability, make sure to run <code class="ph codeph">COMPUTE STATS</code> or <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> on + the table or partition after removing any stale statistics. + </p> + + <p class="p"> + Dropping the statistics is not required for an unpartitioned table or a partitioned table covered by the + original type of statistics. A subsequent <code class="ph codeph">COMPUTE STATS</code> statement replaces any existing + statistics with new ones, for all partitions, regardless of whether the old ones were outdated. Therefore, + this statement was rarely used before the introduction of incremental statistics. + </p> + + <p class="p"> + Dropping the statistics is required for a partitioned table containing incremental statistics, to make a + subsequent <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement rescan an existing partition. See + <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a> for information about incremental statistics, a new feature + available in Impala 2.1.0 and higher. + </p> + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + The user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, does not need any + particular HDFS permissions to perform this statement. + All read and write operations are on the metastore database, + not HDFS files and directories. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <p class="p"> + The following example shows a partitioned table that has associated statistics produced by the + <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement, and how the situation evolves as statistics are dropped + from specific partitions, then the entire table. + </p> + + <p class="p"> + Initially, all table and column statistics are filled in. + </p> + + + +<pre class="pre codeblock"><code>show table stats item_partitioned; ++-------------+-------+--------+----------+--------------+---------+----------------- +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+----------------- +| Books | 1733 | 1 | 223.74KB | NOT CACHED | PARQUET | true +| Children | 1786 | 1 | 230.05KB | NOT CACHED | PARQUET | true +| Electronics | 1812 | 1 | 232.67KB | NOT CACHED | PARQUET | true +| Home | 1807 | 1 | 232.56KB | NOT CACHED | PARQUET | true +| Jewelry | 1740 | 1 | 223.72KB | NOT CACHED | PARQUET | true +| Men | 1811 | 1 | 231.25KB | NOT CACHED | PARQUET | true +| Music | 1860 | 1 | 237.90KB | NOT CACHED | PARQUET | true +| Shoes | 1835 | 1 | 234.90KB | NOT CACHED | PARQUET | true +| Sports | 1783 | 1 | 227.97KB | NOT CACHED | PARQUET | true +| Women | 1790 | 1 | 226.27KB | NOT CACHED | PARQUET | true +| Total | 17957 | 10 | 2.25MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+----------------- +show column stats item_partitioned; ++------------------+-----------+------------------+--------+----------+-------------- +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size ++------------------+-----------+------------------+--------+----------+-------------- +| i_item_sk | INT | 19443 | -1 | 4 | 4 +| i_item_id | STRING | 9025 | -1 | 16 | 16 +| i_rec_start_date | TIMESTAMP | 4 | -1 | 16 | 16 +| i_rec_end_date | TIMESTAMP | 3 | -1 | 16 | 16 +| i_item_desc | STRING | 13330 | -1 | 200 | 100.302803039 +| i_current_price | FLOAT | 2807 | -1 | 4 | 4 +| i_wholesale_cost | FLOAT | 2105 | -1 | 4 | 4 +| i_brand_id | INT | 965 | -1 | 4 | 4 +| i_brand | STRING | 725 | -1 | 22 | 16.1776008605 +| i_class_id | INT | 16 | -1 | 4 | 4 +| i_class | STRING | 101 | -1 | 15 | 7.76749992370 +| i_category_id | INT | 10 | -1 | 4 | 4 +| i_manufact_id | INT | 1857 | -1 | 4 | 4 +| i_manufact | STRING | 1028 | -1 | 15 | 11.3295001983 +| i_size | STRING | 8 | -1 | 11 | 4.33459997177 +| i_formulation | STRING | 12884 | -1 | 20 | 19.9799995422 +| i_color | STRING | 92 | -1 | 10 | 5.38089990615 +| i_units | STRING | 22 | -1 | 7 | 4.18690013885 +| i_container | STRING | 2 | -1 | 7 | 6.99259996414 +| i_manager_id | INT | 105 | -1 | 4 | 4 +| i_product_name | STRING | 19094 | -1 | 25 | 18.0233001708 +| i_category | STRING | 10 | 0 | -1 | -1 ++------------------+-----------+------------------+--------+----------+-------------- +</code></pre> + + <p class="p"> + To remove statistics for particular partitions, use the <code class="ph codeph">DROP INCREMENTAL STATS</code> statement. + After removing statistics for two partitions, the table-level statistics reflect that change in the + <code class="ph codeph">#Rows</code> and <code class="ph codeph">Incremental stats</code> fields. The counts, maximums, and averages of + the column-level statistics are unaffected. + </p> + + <div class="note note note_note"><span class="note__title notetitle">Note:</span> + (It is possible that the row count might be preserved in future after a <code class="ph codeph">DROP INCREMENTAL + STATS</code> statement. Check the resolution of the issue + <a class="xref" href="https://issues.apache.org/jira/browse/IMPALA-1615"; target="_blank">IMPALA-1615</a>.) + </div> + +<pre class="pre codeblock"><code>drop incremental stats item_partitioned partition (i_category='Sports'); +drop incremental stats item_partitioned partition (i_category='Electronics'); + +show table stats item_partitioned ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+----------------- +| Books | 1733 | 1 | 223.74KB | NOT CACHED | PARQUET | true +| Children | 1786 | 1 | 230.05KB | NOT CACHED | PARQUET | true +| Electronics | -1 | 1 | 232.67KB | NOT CACHED | PARQUET | false +| Home | 1807 | 1 | 232.56KB | NOT CACHED | PARQUET | true +| Jewelry | 1740 | 1 | 223.72KB | NOT CACHED | PARQUET | true +| Men | 1811 | 1 | 231.25KB | NOT CACHED | PARQUET | true +| Music | 1860 | 1 | 237.90KB | NOT CACHED | PARQUET | true +| Shoes | 1835 | 1 | 234.90KB | NOT CACHED | PARQUET | true +| Sports | -1 | 1 | 227.97KB | NOT CACHED | PARQUET | false +| Women | 1790 | 1 | 226.27KB | NOT CACHED | PARQUET | true +| Total | 17957 | 10 | 2.25MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+----------------- +show column stats item_partitioned ++------------------+-----------+------------------+--------+----------+-------------- +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size ++------------------+-----------+------------------+--------+----------+-------------- +| i_item_sk | INT | 19443 | -1 | 4 | 4 +| i_item_id | STRING | 9025 | -1 | 16 | 16 +| i_rec_start_date | TIMESTAMP | 4 | -1 | 16 | 16 +| i_rec_end_date | TIMESTAMP | 3 | -1 | 16 | 16 +| i_item_desc | STRING | 13330 | -1 | 200 | 100.302803039 +| i_current_price | FLOAT | 2807 | -1 | 4 | 4 +| i_wholesale_cost | FLOAT | 2105 | -1 | 4 | 4 +| i_brand_id | INT | 965 | -1 | 4 | 4 +| i_brand | STRING | 725 | -1 | 22 | 16.1776008605 +| i_class_id | INT | 16 | -1 | 4 | 4 +| i_class | STRING | 101 | -1 | 15 | 7.76749992370 +| i_category_id | INT | 10 | -1 | 4 | 4 +| i_manufact_id | INT | 1857 | -1 | 4 | 4 +| i_manufact | STRING | 1028 | -1 | 15 | 11.3295001983 +| i_size | STRING | 8 | -1 | 11 | 4.33459997177 +| i_formulation | STRING | 12884 | -1 | 20 | 19.9799995422 +| i_color | STRING | 92 | -1 | 10 | 5.38089990615 +| i_units | STRING | 22 | -1 | 7 | 4.18690013885 +| i_container | STRING | 2 | -1 | 7 | 6.99259996414 +| i_manager_id | INT | 105 | -1 | 4 | 4 +| i_product_name | STRING | 19094 | -1 | 25 | 18.0233001708 +| i_category | STRING | 10 | 0 | -1 | -1 ++------------------+-----------+------------------+--------+----------+-------------- +</code></pre> + + <p class="p"> + To remove all statistics from the table, whether produced by <code class="ph codeph">COMPUTE STATS</code> or + <code class="ph codeph">COMPUTE INCREMENTAL STATS</code>, use the <code class="ph codeph">DROP STATS</code> statement without the + <code class="ph codeph">INCREMENTAL</code> clause). Now, both table-level and column-level statistics are reset. + </p> + +<pre class="pre codeblock"><code>drop stats item_partitioned; + +show table stats item_partitioned ++-------------+-------+--------+----------+--------------+---------+------------------ +| i_category | #Rows | #Files | Size | Bytes Cached | Format | Incremental stats ++-------------+-------+--------+----------+--------------+---------+------------------ +| Books | -1 | 1 | 223.74KB | NOT CACHED | PARQUET | false +| Children | -1 | 1 | 230.05KB | NOT CACHED | PARQUET | false +| Electronics | -1 | 1 | 232.67KB | NOT CACHED | PARQUET | false +| Home | -1 | 1 | 232.56KB | NOT CACHED | PARQUET | false +| Jewelry | -1 | 1 | 223.72KB | NOT CACHED | PARQUET | false +| Men | -1 | 1 | 231.25KB | NOT CACHED | PARQUET | false +| Music | -1 | 1 | 237.90KB | NOT CACHED | PARQUET | false +| Shoes | -1 | 1 | 234.90KB | NOT CACHED | PARQUET | false +| Sports | -1 | 1 | 227.97KB | NOT CACHED | PARQUET | false +| Women | -1 | 1 | 226.27KB | NOT CACHED | PARQUET | false +| Total | -1 | 10 | 2.25MB | 0B | | ++-------------+-------+--------+----------+--------------+---------+------------------ +show column stats item_partitioned ++------------------+-----------+------------------+--------+----------+----------+ +| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size | ++------------------+-----------+------------------+--------+----------+----------+ +| i_item_sk | INT | -1 | -1 | 4 | 4 | +| i_item_id | STRING | -1 | -1 | -1 | -1 | +| i_rec_start_date | TIMESTAMP | -1 | -1 | 16 | 16 | +| i_rec_end_date | TIMESTAMP | -1 | -1 | 16 | 16 | +| i_item_desc | STRING | -1 | -1 | -1 | -1 | +| i_current_price | FLOAT | -1 | -1 | 4 | 4 | +| i_wholesale_cost | FLOAT | -1 | -1 | 4 | 4 | +| i_brand_id | INT | -1 | -1 | 4 | 4 | +| i_brand | STRING | -1 | -1 | -1 | -1 | +| i_class_id | INT | -1 | -1 | 4 | 4 | +| i_class | STRING | -1 | -1 | -1 | -1 | +| i_category_id | INT | -1 | -1 | 4 | 4 | +| i_manufact_id | INT | -1 | -1 | 4 | 4 | +| i_manufact | STRING | -1 | -1 | -1 | -1 | +| i_size | STRING | -1 | -1 | -1 | -1 | +| i_formulation | STRING | -1 | -1 | -1 | -1 | +| i_color | STRING | -1 | -1 | -1 | -1 | +| i_units | STRING | -1 | -1 | -1 | -1 | +| i_container | STRING | -1 | -1 | -1 | -1 | +| i_manager_id | INT | -1 | -1 | 4 | 4 | +| i_product_name | STRING | -1 | -1 | -1 | -1 | +| i_category | STRING | 10 | 0 | -1 | -1 | ++------------------+-----------+------------------+--------+----------+----------+ +</code></pre> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_compute_stats.html#compute_stats">COMPUTE STATS Statement</a>, <a class="xref" href="impala_show.html#show_table_stats">SHOW TABLE STATS Statement</a>, + <a class="xref" href="impala_show.html#show_column_stats">SHOW COLUMN STATS Statement</a>, <a class="xref" href="impala_perf_stats.html#perf_stats">Table and Column Statistics</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_table.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_table.html b/docs/build/html/topics/impala_drop_table.html new file mode 100644 index 0000000..b9d21bc --- /dev/null +++ b/docs/build/html/topics/impala_drop_table.html @@ -0,0 +1,192 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_table"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP TABLE Statement</title></head><body id="drop_table"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP TABLE Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Removes an Impala table. Also removes the underlying HDFS data files for internal tables, although not for + external tables. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>DROP TABLE [IF EXISTS] [<var class="keyword varname">db_name</var>.]<var class="keyword varname">table_name</var> <span class="ph">[PURGE]</span></code></pre> + + <p class="p"> + <strong class="ph b">IF EXISTS clause:</strong> + </p> + + <p class="p"> + The optional <code class="ph codeph">IF EXISTS</code> clause makes the statement succeed whether or not the table exists. + If the table does exist, it is dropped; if it does not exist, the statement has no effect. This capability is + useful in standardized setup scripts that remove existing schema objects and create new ones. By using some + combination of <code class="ph codeph">IF EXISTS</code> for the <code class="ph codeph">DROP</code> statements and <code class="ph codeph">IF NOT + EXISTS</code> clauses for the <code class="ph codeph">CREATE</code> statements, the script can run successfully the first + time you run it (when the objects do not exist yet) and subsequent times (when some or all of the objects do + already exist). + </p> + + <p class="p"> + <strong class="ph b">PURGE clause:</strong> + </p> + + <p class="p"> The optional <code class="ph codeph">PURGE</code> keyword, available in + <span class="keyword">Impala 2.3</span> and higher, causes Impala to remove the associated + HDFS data files immediately, rather than going through the HDFS trashcan + mechanism. Use this keyword when dropping a table if it is crucial to + remove the data as quickly as possible to free up space, or if there is a + problem with the trashcan, such as the trash cannot being configured or + being in a different HDFS encryption zone than the data files. </p> + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> + </p> + + <p class="p"> + By default, Impala removes the associated HDFS directory and data files for the table. If you issue a + <code class="ph codeph">DROP TABLE</code> and the data files are not deleted, it might be for the following reasons: + </p> + + <ul class="ul"> + <li class="li"> + If the table was created with the + <code class="ph codeph"><a class="xref" href="impala_tables.html#external_tables">EXTERNAL</a></code> clause, Impala leaves all + files and directories untouched. Use external tables when the data is under the control of other Hadoop + components, and Impala is only used to query the data files from their original locations. + </li> + + <li class="li"> + Impala might leave the data files behind unintentionally, if there is no HDFS location available to hold + the HDFS trashcan for the <code class="ph codeph">impala</code> user. See + <a class="xref" href="impala_prereqs.html#prereqs_account">User Account Requirements</a> for the procedure to set up the required HDFS home + directory. + </li> + </ul> + + <p class="p"> + Make sure that you are in the correct database before dropping a table, either by issuing a + <code class="ph codeph">USE</code> statement first or by using a fully qualified name + <code class="ph codeph"><var class="keyword varname">db_name</var>.<var class="keyword varname">table_name</var></code>. + </p> + + <p class="p"> + If you intend to issue a <code class="ph codeph">DROP DATABASE</code> statement, first issue <code class="ph codeph">DROP TABLE</code> + statements to remove all the tables in that database. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + +<pre class="pre codeblock"><code>create database temporary; +use temporary; +create table unimportant (x int); +create table trivial (s string); +-- Drop a table in the current database. +drop table unimportant; +-- Switch to a different database. +use default; +-- To drop a table in a different database... +drop table trivial; +<em class="ph i">ERROR: AnalysisException: Table does not exist: default.trivial</em> +-- ...use a fully qualified name. +drop table temporary.trivial;</code></pre> + + <p class="p"> + For other tips about managing and reclaiming Impala disk space, see + <a class="xref" href="../shared/../topics/impala_disk_space.html#disk_space">Managing Disk Space for Impala Data</a>. + </p> + + <p class="p"> + <strong class="ph b">Amazon S3 considerations:</strong> + </p> + <p class="p"> + The <code class="ph codeph">DROP TABLE</code> statement can remove data files from S3 + if the associated S3 table is an internal table. + In <span class="keyword">Impala 2.6</span> and higher, as part of improved support for writing + to S3, Impala also removes the associated folder when dropping an internal table + that resides on S3. + See <a class="xref" href="impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about working with S3 tables. + </p> + + <div class="p"> + For best compatibility with the S3 write support in <span class="keyword">Impala 2.6</span> + and higher: + <ul class="ul"> + <li class="li">Use native Hadoop techniques to create data files in S3 for querying through Impala.</li> + <li class="li">Use the <code class="ph codeph">PURGE</code> clause of <code class="ph codeph">DROP TABLE</code> when dropping internal (managed) tables.</li> + </ul> + By default, when you drop an internal (managed) table, the data files are + moved to the HDFS trashcan. This operation is expensive for tables that + reside on the Amazon S3 filesystem. Therefore, for S3 tables, prefer to use + <code class="ph codeph">DROP TABLE <var class="keyword varname">table_name</var> PURGE</code> rather than the default <code class="ph codeph">DROP TABLE</code> statement. + The <code class="ph codeph">PURGE</code> clause makes Impala delete the data files immediately, + skipping the HDFS trashcan. + For the <code class="ph codeph">PURGE</code> clause to work effectively, you must originally create the + data files on S3 using one of the tools from the Hadoop ecosystem, such as + <code class="ph codeph">hadoop fs -cp</code>, or <code class="ph codeph">INSERT</code> in Impala or Hive. + </div> + + <p class="p"> + In <span class="keyword">Impala 2.6</span> and higher, Impala DDL statements such as + <code class="ph codeph">CREATE DATABASE</code>, <code class="ph codeph">CREATE TABLE</code>, <code class="ph codeph">DROP DATABASE CASCADE</code>, + <code class="ph codeph">DROP TABLE</code>, and <code class="ph codeph">ALTER TABLE [ADD|DROP] PARTITION</code> can create or remove folders + as needed in the Amazon S3 system. Prior to <span class="keyword">Impala 2.6</span>, you had to create folders yourself and point + Impala database, tables, or partitions at them, and manually remove folders when no longer needed. + See <a class="xref" href="../shared/../topics/impala_s3.html#s3">Using Impala with the Amazon S3 Filesystem</a> for details about reading and writing S3 data with Impala. + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> + </p> + <p class="p"> + For an internal table, the user ID that the <span class="keyword cmdname">impalad</span> daemon runs under, + typically the <code class="ph codeph">impala</code> user, must have write + permission for all the files and directories that make up the table. + </p> + <p class="p"> + For an external table, dropping the table only involves changes to metadata in the metastore database. + Because Impala does not remove any HDFS files or directories when external tables are dropped, + no particular permissions are needed for the associated HDFS files or directories. + </p> + + <p class="p"> + <strong class="ph b">Kudu considerations:</strong> + </p> + <p class="p"> + Kudu tables can be managed or external, the same as with HDFS-based + tables. For a managed table, the underlying Kudu table and its data + are removed by <code class="ph codeph">DROP TABLE</code>. For an external table, + the underlying Kudu table and its data remain after a + <code class="ph codeph">DROP TABLE</code>. + </p> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_tables.html#tables">Overview of Impala Tables</a>, + <a class="xref" href="impala_alter_table.html#alter_table">ALTER TABLE Statement</a>, <a class="xref" href="impala_create_table.html#create_table">CREATE TABLE Statement</a>, + <a class="xref" href="impala_partitioning.html#partitioning">Partitioning for Impala Tables</a>, <a class="xref" href="impala_tables.html#internal_tables">Internal Tables</a>, + <a class="xref" href="impala_tables.html#external_tables">External Tables</a> + </p> + + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_drop_view.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_drop_view.html b/docs/build/html/topics/impala_drop_view.html new file mode 100644 index 0000000..123c376 --- /dev/null +++ b/docs/build/html/topics/impala_drop_view.html @@ -0,0 +1,80 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_langref_sql.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="drop_view"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>DROP VIEW Statement</title></head><body id="drop_view"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">DROP VIEW Statement</h1> + + + + <div class="body conbody"> + + <p class="p"> + + Removes the specified view, which was originally created by the <code class="ph codeph">CREATE VIEW</code> statement. + Because a view is purely a logical construct (an alias for a query) with no physical data behind it, + <code class="ph codeph">DROP VIEW</code> only involves changes to metadata in the metastore database, not any data files in + HDFS. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>DROP VIEW [IF EXISTS] [<var class="keyword varname">db_name</var>.]<var class="keyword varname">view_name</var></code></pre> + + <p class="p"> + <strong class="ph b">Statement type:</strong> DDL + </p> + + <p class="p"> + <strong class="ph b">Cancellation:</strong> Cannot be cancelled. + </p> + + <p class="p"> + <strong class="ph b">HDFS permissions:</strong> This statement does not touch any HDFS files or directories, + therefore no HDFS permissions are required. + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <div class="p"> + The following example creates a series of views and then drops them. These examples illustrate how views + are associated with a particular database, and both the view definitions and the view names for + <code class="ph codeph">CREATE VIEW</code> and <code class="ph codeph">DROP VIEW</code> can refer to a view in the current database or + a fully qualified view name. +<pre class="pre codeblock"><code> +-- Create and drop a view in the current database. +CREATE VIEW few_rows_from_t1 AS SELECT * FROM t1 LIMIT 10; +DROP VIEW few_rows_from_t1; + +-- Create and drop a view referencing a table in a different database. +CREATE VIEW table_from_other_db AS SELECT x FROM db1.foo WHERE x IS NOT NULL; +DROP VIEW table_from_other_db; + +USE db1; +-- Create a view in a different database. +CREATE VIEW db2.v1 AS SELECT * FROM db2.foo; +-- Switch into the other database and drop the view. +USE db2; +DROP VIEW v1; + +USE db1; +-- Create a view in a different database. +CREATE VIEW db2.v1 AS SELECT * FROM db2.foo; +-- Drop a view in the other database. +DROP VIEW db2.v1; +</code></pre> + </div> + + <p class="p"> + <strong class="ph b">Related information:</strong> + </p> + + <p class="p"> + <a class="xref" href="impala_views.html#views">Overview of Impala Views</a>, <a class="xref" href="impala_create_view.html#create_view">CREATE VIEW Statement</a>, + <a class="xref" href="impala_alter_view.html#alter_view">ALTER VIEW Statement</a> + </p> + </div> +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_langref_sql.html">Impala SQL Statements</a></div></div></nav></article></main></body></html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/75c46918/docs/build/html/topics/impala_exec_single_node_rows_threshold.html ---------------------------------------------------------------------- diff --git a/docs/build/html/topics/impala_exec_single_node_rows_threshold.html b/docs/build/html/topics/impala_exec_single_node_rows_threshold.html new file mode 100644 index 0000000..aa43a0c --- /dev/null +++ b/docs/build/html/topics/impala_exec_single_node_rows_threshold.html @@ -0,0 +1,89 @@ +<!DOCTYPE html + SYSTEM "about:legacy-compat"> +<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta charset="UTF-8"><meta name="copyright" content="(C) Copyright 2017"><meta name="DC.rights.owner" content="(C) Copyright 2017"><meta name="DC.Type" content="concept"><meta name="DC.Relation" scheme="URI" content="../topics/impala_query_options.html"><meta name="prodname" content="Impala"><meta name="prodname" content="Impala"><meta name="version" content="Impala 2.8.x"><meta name="version" content="Impala 2.8.x"><meta name="DC.Format" content="XHTML"><meta name="DC.Identifier" content="exec_single_node_rows_threshold"><link rel="stylesheet" type="text/css" href="../commonltr.css"><title>EXEC_SINGLE_NODE_ROWS_THRESHOLD Query Option (Impala 2.1 or higher only)</title></head><body id="exec_single_node_rows_threshold"><main role="main"><article role="article" aria-labelledby="ariaid-title1"> + + <h1 class="title topictitle1" id="ariaid-title1">EXEC_SINGLE_NODE_ROWS_THRESHOLD Query Option (<span class="keyword">Impala 2.1</span> or higher only)</h1> + + + + <div class="body conbody"> + + <p class="p"> + + This setting controls the cutoff point (in terms of number of rows scanned) below which Impala treats a query + as a <span class="q">"small"</span> query, turning off optimizations such as parallel execution and native code generation. The + overhead for these optimizations is applicable for queries involving substantial amounts of data, but it + makes sense to skip them for queries involving tiny amounts of data. Reducing the overhead for small queries + allows Impala to complete them more quickly, keeping YARN resources, admission control slots, and so on + available for data-intensive queries. + </p> + + <p class="p"> + <strong class="ph b">Syntax:</strong> + </p> + +<pre class="pre codeblock"><code>SET EXEC_SINGLE_NODE_ROWS_THRESHOLD=<var class="keyword varname">number_of_rows</var></code></pre> + + <p class="p"> + <strong class="ph b">Type:</strong> numeric + </p> + + <p class="p"> + <strong class="ph b">Default:</strong> 100 + </p> + + <p class="p"> + <strong class="ph b">Usage notes:</strong> Typically, you increase the default value to make this optimization apply to more queries. + If incorrect or corrupted table and column statistics cause Impala to apply this optimization + incorrectly to queries that actually involve substantial work, you might see the queries being slower as a + result of remote reads. In that case, recompute statistics with the <code class="ph codeph">COMPUTE STATS</code> + or <code class="ph codeph">COMPUTE INCREMENTAL STATS</code> statement. If there is a problem collecting accurate + statistics, you can turn this feature off by setting the value to -1. + </p> + + <p class="p"> + <strong class="ph b">Internal details:</strong> + </p> + + <p class="p"> + This setting applies to query fragments where the amount of data to scan can be accurately determined, either + through table and column statistics, or by the presence of a <code class="ph codeph">LIMIT</code> clause. If Impala cannot + accurately estimate the size of the input data, this setting does not apply. + </p> + + <p class="p"> + In <span class="keyword">Impala 2.3</span> and higher, where Impala supports the complex data types <code class="ph codeph">STRUCT</code>, + <code class="ph codeph">ARRAY</code>, and <code class="ph codeph">MAP</code>, if a query refers to any column of those types, + the small-query optimization is turned off for that query regardless of the + <code class="ph codeph">EXEC_SINGLE_NODE_ROWS_THRESHOLD</code> setting. + </p> + + <p class="p"> + For a query that is determined to be <span class="q">"small"</span>, all work is performed on the coordinator node. This might + result in some I/O being performed by remote reads. The savings from not distributing the query work and not + generating native code are expected to outweigh any overhead from the remote reads. + </p> + + <p class="p"> + <strong class="ph b">Added in:</strong> <span class="keyword">Impala 2.1.0</span> + </p> + + <p class="p"> + <strong class="ph b">Examples:</strong> + </p> + + <p class="p"> + A common use case is to query just a few rows from a table to inspect typical data values. In this example, + Impala does not parallelize the query or perform native code generation because the result set is guaranteed + to be smaller than the threshold value from this query option: + </p> + +<pre class="pre codeblock"><code>SET EXEC_SINGLE_NODE_ROWS_THRESHOLD=500; +SELECT * FROM enormous_table LIMIT 300; +</code></pre> + + + + </div> + +<nav role="navigation" class="related-links"><div class="familylinks"><div class="parentlink"><strong>Parent topic:</strong> <a class="link" href="../topics/impala_query_options.html">Query Options for the SET Statement</a></div></div></nav></article></main></body></html> \ No newline at end of file
