http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/catalog/pg_tablespace.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/catalog/pg_tablespace.html.md.erb b/markdown/reference/catalog/pg_tablespace.html.md.erb new file mode 100644 index 0000000..493c6b0 --- /dev/null +++ b/markdown/reference/catalog/pg_tablespace.html.md.erb @@ -0,0 +1,22 @@ +--- +title: pg_tablespace +--- + +The `pg_tablespace` system catalog table stores information about the available tablespaces. Tables can be placed in particular tablespaces to aid administration of disk layout. Unlike most system catalogs, `pg_tablespace` is shared across all databases of a HAWQ system: there is only one copy of `pg_tablespace` per system, not one per database. + +<a id="topic1__hx156260"></a> +<span class="tablecap">Table 1. pg\_catalog.pg\_tablespace</span> + +| column | type | references | description | +|-------------------|-------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------| +| `spcname` | name | Â | Tablespace name. | +| `spcowner` | oid | pg\_authid.oid | Owner of the tablespace, usually the user who created it. | +| `spclocation` | text\[\] | Â | Deprecated. | +| `spcacl ` | aclitem\[\] | Â | Tablespace access privileges. | +| `spcprilocations` | text\[\] | Â | Deprecated. | +| `spcmrilocations` | text\[\] | Â | Deprecated. | +| `spcfsoid` | oid | pg\_filespace.oid | The object id of the filespace used by this tablespace. A filespace defines directory locations on the master and segments. | + + + +
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/catalog/pg_trigger.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/catalog/pg_trigger.html.md.erb b/markdown/reference/catalog/pg_trigger.html.md.erb new file mode 100644 index 0000000..3074e46 --- /dev/null +++ b/markdown/reference/catalog/pg_trigger.html.md.erb @@ -0,0 +1,114 @@ +--- +title: pg_trigger +--- + +The `pg_trigger` system catalog table stores triggers on tables. + +**Note:** HAWQ does not support triggers. + +<a id="topic1__hy183441"></a> +<span class="tablecap">Table 1. pg\_catalog.pg\_trigger</span> + +<table> +<colgroup> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +</colgroup> +<thead> +<tr class="header"> +<th>column</th> +<th>type</th> +<th>references</th> +<th>description</th> +</tr> +</thead> +<tbody> +<tr class="odd"> +<td><code class="ph codeph">tgrelid</code></td> +<td>oid</td> +<td><em>pg_class.oid</em> +<p>Note that HAWQ does not enforce referential integrity.</p></td> +<td>The table this trigger is on.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tgname</code></td> +<td>name</td> +<td> </td> +<td>Trigger name (must be unique among triggers of same table).</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgfoid</code></td> +<td>oid</td> +<td><em>pg_proc.oid</em> +<p>Note that HAWQ does not enforce referential integrity.</p></td> +<td>The function to be called.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tgtype</code></td> +<td>smallint</td> +<td> </td> +<td>Bit mask identifying trigger conditions.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgenabled</code></td> +<td>boolean</td> +<td> </td> +<td>True if trigger is enabled.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tgisconstraint</code></td> +<td>boolean</td> +<td> </td> +<td>True if trigger implements a referential integrity constraint.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgconstrname</code></td> +<td>name</td> +<td> </td> +<td>Referential integrity constraint name.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tgconstrrelid</code></td> +<td>oid</td> +<td><em>pg_class.oid</em> +<p>Note that HAWQ does not enforce referential integrity.</p></td> +<td>The table referenced by an referential integrity constraint.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgdeferrable</code></td> +<td>boolean</td> +<td> </td> +<td>True if deferrable.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tginitdeferred</code></td> +<td>boolean</td> +<td> </td> +<td>True if initially deferred.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgnargs</code></td> +<td>smallint</td> +<td> </td> +<td>Number of argument strings passed to trigger function.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">tgattr</code></td> +<td>int2vector</td> +<td> </td> +<td>Currently not used.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">tgargs</code></td> +<td>bytea</td> +<td> </td> +<td>Argument strings to pass to trigger, each NULL-terminated.</td> +</tr> +</tbody> +</table> + + + + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/catalog/pg_type.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/catalog/pg_type.html.md.erb b/markdown/reference/catalog/pg_type.html.md.erb new file mode 100644 index 0000000..e2ea28a --- /dev/null +++ b/markdown/reference/catalog/pg_type.html.md.erb @@ -0,0 +1,176 @@ +--- +title: pg_type +--- + +The `pg_type` system catalog table stores information about data types. Base types (scalar types) are created with `CREATE TYPE`, and domains with `CREATE DOMAIN`. A composite type is automatically created for each table in the database, to represent the row structure of the table. It is also possible to create composite types with `CREATE TYPE AS`. + +<a id="topic1__hz156260"></a> +<span class="tablecap">Table 1. pg\_catalog.pg\_type</span> + +<table> +<colgroup> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +</colgroup> +<thead> +<tr class="header"> +<th>column</th> +<th>type</th> +<th>references</th> +<th>description</th> +</tr> +</thead> +<tbody> +<tr class="odd"> +<td><code class="ph codeph">typname</code></td> +<td>name</td> +<td> </td> +<td>Data type name.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typnamespace</code></td> +<td>oid</td> +<td>pg_namespace.oid</td> +<td>The OID of the namespace that contains this type.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typowner</code></td> +<td>oid</td> +<td>pg_authid.oid</td> +<td>Owner of the type.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typlen</code></td> +<td>smallint</td> +<td> </td> +<td>For a fixed-size type, <code class="ph codeph">typlen</code> is the number of bytes in the internal representation of the type. But for a variable-length type, <code class="ph codeph">typlen</code> is negative. <code class="ph codeph">-1</code> indicates a 'varlena' type (one that has a length word), <code class="ph codeph">-2</code> indicates a null-terminated C string.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typbyval</code></td> +<td>boolean</td> +<td> </td> +<td>Determines whether internal routines pass a value of this type by value or by reference. <code class="ph codeph">typbyval </code>had better be false if <code class="ph codeph">typlen</code> is not 1, 2, or 4 (or 8 on machines where Datum is 8 bytes). Variable-length types are always passed by reference. Note that <code class="ph codeph">typbyval</code> can be false even if the length would allow pass-by-value; this is currently true for type <code class="ph codeph">float4</code>, for example.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typtype</code></td> +<td>char</td> +<td> </td> +<td><code class="ph codeph">b</code> for a base type, <code class="ph codeph">c</code> for a composite type, <code class="ph codeph">d</code> for a domain, or <code class="ph codeph">p</code> for a pseudo-type.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typisdefined</code></td> +<td>boolean</td> +<td> </td> +<td>True if the type is defined, false if this is a placeholder entry for a not-yet-defined type. When false, nothing except the type name, namespace, and OID can be relied on.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typdelim</code></td> +<td>char</td> +<td> </td> +<td>Character that separates two values of this type when parsing array input. Note that the delimiter is associated with the array element data type, not the array data type.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typrelid</code></td> +<td>oid</td> +<td>pg_class.oid</td> +<td>If this is a composite type, then this column points to the <code class="ph codeph">pg_class</code> entry that defines the corresponding table. (For a free-standing composite type, the <code class="ph codeph">pg_class</code> entry does not really represent a table, but it is needed anyway for the type's <code class="ph codeph">pg_attribute</code> entries to link to.) Zero for non-composite types.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typelem</code></td> +<td>oid</td> +<td>pg_type.oid</td> +<td>If not <code class="ph codeph">0</code> then it identifies another row in pg_type. The current type can then be subscripted like an array yielding values of type <code class="ph codeph">typelem</code>. A true array type is variable length (<code class="ph codeph">typlen</code> = <code class="ph codeph">-1</code>), but some fixed-length (<code class="ph codeph">tylpen</code> > <code class="ph codeph">0</code>) types also have nonzero <code class="ph codeph">typelem</code>, for example <code class="ph codeph">name</code> and <code class="ph codeph">point</code>. If a fixed-length type has a <code class="ph codeph">typelem</code> then its internal representation must be some number of values of the <code class="ph codeph">typelem</code> data type with no other data. Variable-length array types have a header defined by the array subroutines.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typinput</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>Input conversion function (text format).</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typoutput</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>Output conversion function (text format).</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typreceive</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>Input conversion function (binary format), or 0 if none.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typsend</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>Output conversion function (binary format), or 0 if none.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typanalyze</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>Custom <code class="ph codeph">ANALYZE</code> function, or 0 to use the standard function.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typalign</code></td> +<td>char</td> +<td> </td> +<td>The alignment required when storing a value of this type. It applies to storage on disk as well as most representations of the value inside HAWQ. When multiple values are stored consecutively, such as in the representation of a complete row on disk, padding is inserted before a datum of this type so that it begins on the specified boundary. The alignment reference is the beginning of the first datum in the sequence. Possible values are: +<p><code class="ph codeph">c</code> = char alignment (no alignment needed).</p> +<p><code class="ph codeph">s</code> = short alignment (2 bytes on most machines).</p> +<p><code class="ph codeph">i</code> = int alignment (4 bytes on most machines).</p> +<p><code class="ph codeph">d</code> = double alignment (8 bytes on many machines, but not all).</p></td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typstorage</code></td> +<td>char</td> +<td> </td> +<td>For varlena types (those with <code class="ph codeph">typlen</code> = -1) tells if the type is prepared for toasting and what the default strategy for attributes of this type should be. Possible values are: +<p><code class="ph codeph">p</code>: Value must always be stored plain.</p> +<p><code class="ph codeph">e</code>: Value can be stored in a secondary relation (if relation has one, see <code class="ph codeph">pg_class.reltoastrelid</code>).</p> +<p><code class="ph codeph">m</code>: Value can be stored compressed inline.</p> +<p><code class="ph codeph">x</code>: Value can be stored compressed inline or stored in secondary storage.</p> +<p>Note that <code class="ph codeph">m</code> columns can also be moved out to secondary storage, but only as a last resort (<code class="ph codeph">e</code> and <code class="ph codeph">x</code> columns are moved first).</p></td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typnotnull</code></td> +<td>boolean</td> +<td> </td> +<td>Represents a not-null constraint on a type. Used for domains only.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typbasetype</code></td> +<td>oid</td> +<td>pg_type.oid</td> +<td>Identifies the type that a domain is based on. Zero if this type is not a domain.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typtypmod</code></td> +<td>integer</td> +<td> </td> +<td>Domains use typtypmod to record the typmod to be applied to their base type (-1 if base type does not use a typmod). -1 if this type is not a domain.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typndims</code></td> +<td>integer</td> +<td> </td> +<td>The number of array dimensions for a domain that is an array (if <code class="ph codeph">typbasetype</code> is an array type; the domain's <code class="ph codeph">typelem</code> will match the base type's <code class="ph codeph">typelem</code>). Zero for types other than array domains.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">typdefaultbin</code></td> +<td>text</td> +<td> </td> +<td>If not null, it is the <code class="ph codeph">nodeToString()</code> representation of a default expression for the type. This is only used for domains.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">typdefault</code></td> +<td>text</td> +<td> </td> +<td>Null if the type has no associated default value. If not null, typdefault must contain a human-readable version of the default expression represented by typdefaultbin. If typdefaultbin is null and typdefault is not, then typdefault is the external representation of the type's default value, which may be fed to the type's input converter to produce a constant.</td> +</tr> +</tbody> +</table> + + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/catalog/pg_type_encoding.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/catalog/pg_type_encoding.html.md.erb b/markdown/reference/catalog/pg_type_encoding.html.md.erb new file mode 100644 index 0000000..b38ff10 --- /dev/null +++ b/markdown/reference/catalog/pg_type_encoding.html.md.erb @@ -0,0 +1,15 @@ +--- +title: pg_type_encoding +--- + +The `pg_type_encoding` system catalog table contains the column storage type information. + +<a id="topic1__ia177831"></a> +<span class="tablecap">Table 1. pg\_catalog.pg\_type\_encoding</span> + +| column | type | modifers | storage | description | +|--------------|------------|----------|----------|----------------------------------------------------------------------------------| +| `typeid` | oid | not null | plain | Foreign key to [pg\_attribute](pg_attribute.html#topic1) | +| `typoptions` | text \[ \] |  | extended | The actual options | + + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/catalog/pg_window.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/catalog/pg_window.html.md.erb b/markdown/reference/catalog/pg_window.html.md.erb new file mode 100644 index 0000000..afe4c0d --- /dev/null +++ b/markdown/reference/catalog/pg_window.html.md.erb @@ -0,0 +1,97 @@ +--- +title: pg_window +--- + +The `pg_window` table stores information about window functions. Window functions are often used to compose complex OLAP (online analytical processing) queries. Window functions are applied to partitioned result sets within the scope of a single query expression. A window partition is a subset of rows returned by a query, as defined in a special `OVER()` clause. Typical window functions are `rank`, `dense_rank`, and `row_number`. Each entry in `pg_window` is an extension of an entry in [pg\_proc](pg_proc.html#topic1). The [pg\_proc](pg_proc.html#topic1) entry carries the window function's name, input and output data types, and other information that is similar to ordinary functions. + +<a id="topic1__ic143898"></a> +<span class="tablecap">Table 1. pg\_catalog.pg\_window</span> + +<table> +<colgroup> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +<col width="25%" /> +</colgroup> +<thead> +<tr class="header"> +<th>column</th> +<th>type</th> +<th>references</th> +<th>description</th> +</tr> +</thead> +<tbody> +<tr class="odd"> +<td><code class="ph codeph">winfnoid</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>The OID in <code class="ph codeph">pg_proc</code> of the window function.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">winrequireorder</code></td> +<td>boolean</td> +<td> </td> +<td>The window function requires its window specification to have an <code class="ph codeph">ORDER BY</code> clause.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">winallowframe</code></td> +<td>boolean</td> +<td> </td> +<td>The window function permits its window specification to have a <code class="ph codeph">ROWS</code> or <code class="ph codeph">RANGE</code> framing clause.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">winpeercount</code></td> +<td>boolean</td> +<td> </td> +<td>The peer group row count is required to compute this window function, so the Window node implementation must 'look ahead' as necessary to make this available in its internal state.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">wincount</code></td> +<td>boolean</td> +<td> </td> +<td>The partition row count is required to compute this window function.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">winfunc</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>The OID in <code class="ph codeph">pg_proc</code> of a function to compute the value of an immediate-type window function.</td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">winprefunc</code></td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>The OID in <code class="ph codeph">pg_proc</code> of a preliminary window function to compute the partial value of a deferred-type window function.</td> +</tr> +<tr class="even"> +<td><code class="ph codeph">winpretype</code></td> +<td>oid</td> +<td>pg_type.oid</td> +<td>The OID in <code class="ph codeph">pg_type</code> of the preliminary window function's result type.</td> +</tr> +<tr class="odd"> +<td>winfinfunc</td> +<td>regproc</td> +<td>pg_proc.oid</td> +<td>The OID in <code class="ph codeph">pg_proc</code> of a function to compute the final value of a deferred-type window function from the partition row count and the result of <code class="ph codeph">winprefunc</code>.</td> +</tr> +<tr class="even"> +<td>winkind</td> +<td>char</td> +<td> </td> +<td>A character indicating membership of the window function in a class of related functions: +<p><code class="ph codeph">w</code> - ordinary window functions</p> +<p><code class="ph codeph">n</code> - NTILE functions</p> +<p><code class="ph codeph">f</code> - FIRST_VALUE functions</p> +<p><code class="ph codeph">l</code> - LAST_VALUE functions</p> +<p><code class="ph codeph">g</code> - LAG functions</p> +<p><code class="ph codeph">d</code> - LEAD functions</p></td> +</tr> +</tbody> +</table> + + + + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/analyzedb.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/analyzedb.html.md.erb b/markdown/reference/cli/admin_utilities/analyzedb.html.md.erb new file mode 100644 index 0000000..0384c34 --- /dev/null +++ b/markdown/reference/cli/admin_utilities/analyzedb.html.md.erb @@ -0,0 +1,160 @@ +--- +title: analyzedb +--- + +A utility that performs `ANALYZE` operations on tables incrementally and concurrently. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +analyzedb -d <dbname> -s <schema> + [ --full ] + [ -l | --list ] + [ -p <parallel-level> ] + [ -v | --verbose ] + [ -a ] + +analyzedb -d <dbname> -t <schema>.<table> + [ -i col1[, col2, ...] | -x col1[, col2, ...] ] + [ --full ] + [ -l | --list ] + [ -p <parallel-level> ] + [ -v | --verbose ] + [ -a ] + +analyzedb -d <dbname> -f <config-file> | --file <config-file> + [ --full ] + [ -l | --list ] + [ -p <parallel-level> ] + [ -v | --verbose ] + [ -a ] + +analyzedb -d <dbname> --clean_last | --clean_all + +analyzedb --version + +analyzedb -? | -h | --help +``` + +## <a id="topic1__section3"></a>Description + +The `analyzedb` utility updates statistics on table data for the specified tables in a HAWQ database incrementally and concurrently. + +While performing `ANALYZE` operations, `analyzedb` creates a snapshot of the table metadata and stores it on disk on the master host. An `ANALYZE` operation is performed only if the table has been modified. If a table or partition has not been modified since the last time it was analyzed, `analyzedb` automatically skips the table or partition because it already contains up-to-date statistics. + +For a partitioned table `analyzedb` analyzes only those partitions that have no statistics, or that have stale statistics. `analyzedb` also refreshes the statistics on the root partition. + +By default, `analyzedb` creates a maximum of 5 concurrent sessions to analyze tables in parallel. For each session, `analyzedb` issues an `ANALYZE` command to the database and specifies different table names. The `-p` option controls the maximum number of concurrent sessions. + +## <a id="topic1__section4"></a>Notes + +The utility determines if a table has been modified by comparing catalog metadata of tables with the snapshot of metadata taken during a previous `analyzedb` operation. The snapshots of table metadata are stored as state files in the directory `db_analyze` in the HAWQ master data directory. You can specify the `--clean_last` or `--clean_all` option to remove state files generated by `analyzedb`. + +If you do not specify a table, set of tables, or schema, the `analyzedb` utility collects the statistics as needed on all system catalog tables and user-defined tables in the database. + +External tables are not affected by `analyzedb`. + +Table names that contain spaces are not supported. + + +## <a id="topic1__section5"></a>Arguments + +<dt>-d \<dbname\> </dt> +<dd>Specifies the name of the database that contains the tables to be analyzed. If this option is not specified, the database name is read from the environment variable `PGDATABASE`. If `PGDATABASE` is not set, the user name specified for the connection is used.</dd> + +<dt>-s \<schema\> </dt> +<dd>Specify a schema to analyze. All tables in the schema will be analyzed. Only a single schema name can be specified on the command line. + +Only one of the options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`.</dd> + +<dt>-t \<schema\>.\<table\> </dt> +<dd>Collect statistics only on \<schema\>.\<table\>. The table name must be qualified with a schema name. Only a single table name can be specified on the command line. You can specify the `-f` option to specify multiple tables in a file or the `-s` option to specify all the tables in a schema. + +Only one of these options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`.</dd> + +<dt>-f, -\\\-file \<config-file\> </dt> +<dd>Text file that contains a list of tables to be analyzed. A relative file path from current directory can be specified. + +The file lists one table per line. Table names must be qualified with a schema name. Optionally, a list of columns can be specified using the `-i` or `-x`. No other options are allowed in the file. Other options such as `--full` must be specified on the command line. + +Only one of the options can be used to specify the files to be analyzed: `-f` or `--file`, `-t` , or `-s`. + +When performing `ANALYZE` operations on multiple tables, `analyzedb` creates concurrent sessions to analyze tables in parallel. The `-p` option controls the maximum number of concurrent sessions. + +In the following example, the first line performs an `ANALYZE` operation on the table `public.nation`, the second line performs an `ANALYZE` operation only on the columns `l_shipdate` and `l_receiptdate` in the table `public.lineitem`. + +``` pre +public.nation +public.lineitem -i l_shipdate, l_receiptdate +``` +</dd> + + +## <a id="topic1__section5"></a>Options + + +<dt>-x \<col1\>, \<col2\>, ... </dt> +<dd>Optional. Must be specified with the `-t` option. For the table specified with the `-t` option, exclude statistics collection for the specified columns. Statistics are collected only on the columns that are not listed. + +Only `-i`, or `-x` can be specified. Both options cannot be specified.</dd> + +<dt>-i \<col1\>, \<col2\>, ... </dt> +<dd>Optional. Must be specified with the `-t` option. For the table specified with the `-t` option, collect statistics only for the specified columns. + +Only `-i`, or `-x` can be specified. Both options cannot be specified.</dd> + +<dt>-\\\-full </dt> +<dd>Perform an `ANALYZE` operation on all the specified tables. The operation is performed even if the statistics are up to date.</dd> + +<dt>-l, -\\\-list </dt> +<dd>Lists the tables that would have been analyzed with the specified options. The `ANALYZE` operations are not performed.</dd> + +<dt>-p \<parallel-level\> </dt> +<dd>The number of tables that are analyzed in parallel. The value for <parallel-level> can be an integer between 1 and 10, inclusive. Default value is 5.</dd> + +<dt>-a </dt> +<dd>Quiet mode. Do not prompt for user confirmation.</dd> + +<dt> -v, -\\\-verbose </dt> +<dd>If specified, sets the logging level to verbose. Additional log information is written to the log file and the command line during command execution.</dd> + +<dt>-\\\-clean\_last </dt> +<dd>Remove the state files generated by last `analyzedb` operation. All other options except `-d` are ignored.</dd> + +<dt>-\\\-clean\_all </dt> +<dd>Remove all the state files generated by `analyzedb`. All other options except` -d` are ignored.</dd> + +<dt>-h, -?, -\\\-help </dt> +<dd>Displays the online help.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + + +## <a id="topic1__section6"></a>Examples + +An example that collects statistics only on a set of table columns. In the database `mytest`, collect statistics on the columns `shipdate` and `receiptdate` in the table `public.orders`: + +``` shell +$ analyzedb -d mytest -t public.orders -i shipdate, receiptdate +``` + +An example that collects statistics on a table and exclude a set of columns. In the database `mytest`, collect statistics on the table `public.foo`, and do not collect statistics on the columns `bar` and `test2`. + +``` shell +$ analyzedb -d mytest -t public.foo -x bar, test2 +``` + +An example that specifies a file that contains a list of tables. This command collect statistics on the tables listed in the file `analyze-tables` in the database named `mytest`. + +``` shell +$ analyzedb -d mytest -f analyze-tables +``` + +If you do not specify a table, set of tables, or schema, the `analyzedb` utility collects the statistics as needed on all catalog tables and user-defined tables in the specified database. This command refreshes table statistics on the system catalog tables and user-defined tables in the database `mytest`. + +``` shell +$ analyzedb -d mytest +``` + + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/gpfdist.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/gpfdist.html.md.erb b/markdown/reference/cli/admin_utilities/gpfdist.html.md.erb new file mode 100644 index 0000000..1683ddb --- /dev/null +++ b/markdown/reference/cli/admin_utilities/gpfdist.html.md.erb @@ -0,0 +1,157 @@ +--- +title: gpfdist +--- + +Serves data files to or writes data files out from HAWQ segments. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +gpfdist [-d <directory>] [-p <http_port>] [-l <log_file>] [-t <timeout>] + [-S] [-w <time>] [-v | -V] [-s] [-m <max_length>] [--ssl <certificate_path>] + +gpfdist -? | --help + +gpfdist --version +``` + +## <a id="topic1__section3"></a>Description + +`gpfdist` is HAWQ parallel file distribution program. It is used by readable external tables and `hawq load` to serve external table files to all HAWQ segments in parallel. It is used by writable external tables to accept output streams from HAWQ segments in parallel and write them out to a file. + +In order for `gpfdist` to be used by an external table, the `LOCATION` clause of the external table definition must specify the external table data using the `gpfdist://` protocol (see the HAWQ command `CREATE EXTERNAL TABLE`). + +**Note:** If the `--ssl` option is specified to enable SSL security, create the external table with the `gpfdists://` protocol. + +The benefit of using `gpfdist` is that you are guaranteed maximum parallelism while reading from or writing to external tables, thereby offering the best performance as well as easier administration of external tables. + +For readable external tables, `gpfdist` parses and serves data files evenly to all the segment instances in the HAWQ system when users `SELECT` from the external table. For writable external tables, `gpfdist` accepts parallel output streams from the segments when users `INSERT` into the external table, and writes to an output file. + +For readable external tables, if load files are compressed using `gzip` or `bzip2` (have a `.gz` or `.bz2` file extension), `gpfdist` uncompresses the files automatically before loading provided that `gunzip` or `bunzip2` is in your path. + +**Note:** Currently, readable external tables do not support compression on Windows platforms, and writable external tables do not support compression on any platforms. + +To run `gpfdist` on your ETL machines, refer to [Client-Based HAWQ Load Tools](../../../datamgmt/load/client-loadtools.html) for more information. + +**Note:** When using IPv6, always enclose the numeric IP address in brackets. + +You can also run `gpfdist` as a Windows Service. See [Running gpfdist as a Windows Service](#topic1__section5) for more details. + +## <a id="topic1__section4"></a>Options + +<dt>-d \<directory\> </dt> +<dd>The directory from which `gpfdist` will serve files for readable external tables or create output files for writable external tables. If not specified, defaults to the current directory.</dd> + +<dt>-l \<log\_file\> </dt> +<dd>The fully qualified path and log file name where standard output messages are to be logged.</dd> + +<dt>-p \<http\_port\> </dt> +<dd>The HTTP port on which `gpfdist` will serve files. Defaults to 8080.</dd> + +<dt>-t \<timeout\> </dt> +<dd>Sets the time allowed for HAWQ to establish a connection to a `gpfdist` process. Default is 5 seconds. Allowed values are 2 to 600 seconds. May need to be increased on systems with a lot of network traffic.</dd> + +<dt>-m \<max\_length\> </dt> +<dd>Sets the maximum allowed data row length in bytes. Default is 32768. Should be used when user data includes very wide rows (or when `line too long` error message occurs). Should not be used otherwise as it increases resource allocation. Valid range is 32K to 256MB. (The upper limit is 1MB on Windows systems.)</dd> + +<dt>-s </dt> +<dd>Enables simplified logging. When this option is specified, only messages with `WARN` level and higher are written to the `gpfdist` log file. `INFO` level messages are not written to the log file. If this option is not specified, all `gpfdist` messages are written to the log file. + +You can specify this option to reduce the information written to the log file.</dd> + +<dt>-S (use O\_SYNC) </dt> +<dd>Opens the file for synchronous I/O with the `O_SYNC` flag. Any writes to the resulting file descriptor block `gpfdist` until the data is physically written to the underlying hardware.</dd> + +<dt>-w \<time\> </dt> +<dd>Sets the number of seconds that HAWQ delays before closing a target file such as a named pipe. The default value is 0, no delay. The maximum value is 600 seconds, 10 minutes. + +For a HAWQ with multiple segments, there might be a delay between segments when writing data from different segments to the file. You can specify a time to wait before HAWQ closes the file to ensure all the data is written to the file.</dd> + +<dt>-\\\-ssl \<certificate\_path\> </dt> +<dd>Adds SSL encryption to data transferred with `gpfdist`. After executing `gpfdist` with the `--ssl <certificate_path>` option, the only way to load data from this file server is with the `gpfdist://` protocol. + +The location specified in \<certificate\_path\> must contain the following files: + +- The server certificate file, `server.crt` +- The server private key file, `server.key` +- The trusted certificate authorities, `root.crt` + +The root directory (`/`) cannot be specified as \<certificate\_path\>.</dd> + +<dt>-v (verbose) </dt> +<dd>Verbose mode shows progress and status messages.</dd> + +<dt>-V (very verbose) </dt> +<dd>Verbose mode shows all output messages generated by this utility.</dd> + +<dt>-? (help) </dt> +<dd>Displays the online help.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + +## <a id="topic1__section5"></a>Running gpfdist as a Windows Service + +HAWQ Loaders allow `gpfdist` to run as a Windows Service. + +Follow the instructions below to download, register and activate `gpfdist` as a service: + +1. Update your HAWQ Loaders for Windows package to the latest version. See [HAWQ Loader Tools for Windows](../../../datamgmt/load/client-loadtools.html#installloadrunwin) for install and configuration information. + +2. Register `gpfdist` as a Windows service: + 1. Open a Windows command window + 2. Run the following command: + + ``` pre + sc create gpfdist binpath= "<loader_install_dir>\bin\gpfdist.exe -p 8081 -d \"<external_load_files_path>\" -l \"<log_file_path>\"" + ``` + + You can create multiple instances of `gpfdist` by running the same command again, with a unique name and port number for each instance: + + ``` pre + sc create gpfdistN binpath= "<loader_install_dir>\bin\gpfdist.exe -p 8082 -d \"<external_load_files_path>\" -l \"<log_file_path>\"" + ``` + +3. Activate the `gpfdist` service: + 1. Open the Windows Control Panel and select **Administrative Tools > Services**. + 2. Highlight then right-click on the `gpfdist` service in the list of services. + 3. Select **Properties** from the right-click menu, the Service Properties window opens. + + Note that you can also stop this service from the Service Properties window. + + 4. Optional: Change the **Startup Type** to **Automatic** (after a system restart, this service will be running), then under **Service** status, click **Start**. + 5. Click **OK**. + +Repeat the above steps for each instance of `gpfdist` that you created. + +## <a id="topic1__section6"></a>Examples + +To serve files from a specified directory using port 8081 (and start `gpfdist` in the background): + +``` shell +$ gpfdist -d /var/load_files -p 8081 & +``` + +To start `gpfdist` in the background and redirect output and errors to a log file: + +``` shell +$ gpfdist -d /var/load_files -p 8081 -l /home/gpadmin/log & +``` + +To stop `gpfdist` when it is running in the background: + +--First find its process id: + +``` shell +$ ps ax | grep gpfdist +``` + +--Then kill the process, for example: + +``` shell +$ kill 3456 +``` + +## <a id="topic1__section7"></a>See Also + +[hawq load](hawqload.html#topic1), [CREATE EXTERNAL TABLE](../../sql/CREATE-EXTERNAL-TABLE.html) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/gplogfilter.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/gplogfilter.html.md.erb b/markdown/reference/cli/admin_utilities/gplogfilter.html.md.erb new file mode 100644 index 0000000..44e73c5 --- /dev/null +++ b/markdown/reference/cli/admin_utilities/gplogfilter.html.md.erb @@ -0,0 +1,180 @@ +--- +title: gplogfilter +--- + +Searches through HAWQ log files for specified entries. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +gplogfilter [<timestamp_options>] [<pattern_matching_options>] + [<output_options>] [<input_options>] + +gplogfilter --help + +gplogfilter --version +``` +where: + +``` pre +<timestamp_options> = + [-b <datetime> | --begin <datetime>] + [-e <datetime> | --end <datetime>] + [-d <time> | --duration <time>] + +<pattern_matching_options> = + [-c i[gnore] | r[espect] | --case i[gnore] | r[espect]] + [-C '<string>' | --columns '<string>'] + [-f '<string>' | --find '<string>'] + [-F '<string> | --nofind '<string>'] + [-m <regex> | --match <regex>] + [-M <regex>] | --nomatch <regex>] + [-t | --trouble] + +<output_options> = + [-n <integer> | --tail <integer>] + [-s <offset> [<limit>] | --slice <offset> [<limit>]] + [-o <output_file> | --out <output_file>] + [-z <0..9> | --zip <0..9>] + [-a | --append] + +<input_options> = + [<input_file> [-u | --unzip]] +``` + + +## <a id="topic1__section3"></a>Description + +The `gplogfilter` utility can be used to search through a HAWQ log file for entries matching the specified criteria. To read from standard input, use a dash (`-`) as the input file name. Input files may be compressed using `gzip`. In an input file, a log entry is identified by its timestamp in `YYYY-MM-DD [hh:mm[:ss]]` format. + +You can also use `gplogfilter` to search through all segment log files at once by running it through the [hawq ssh](hawqssh.html#topic1) utility. For example, to display the last three lines of each segment log file: + +``` shell +$ hawq ssh -f seg_hostfile_hawqssh +=> source /usr/local/hawq/greenplum_path.sh +=> gplogfilter -n 3 /data/hawq-install-path/segmentdd/pg_log/hawq*.csv +``` + +By default, the output of `gplogfilter` is sent to standard output. Use the `-o` option to send the output to a file or a directory. If you supply an output file name ending in `.gz`, the output file will be compressed by default using maximum compression. If the output destination is a directory, the output file is given the same name as the input file. + +## <a id="topic1__section4"></a>Options + + +**\<input_options\>** + +<dt>\<input\_file\></dt> +<dd>The name of the input log file(s) to search through. To read from standard input, use a dash (`-`) as the input file name.</dd> + +<dt>-u, -\\\-unzip </dt> +<dd>Uncompress the input file using `gunzip`. If the input file name ends in `.gz`, it will be uncompressed by default.</dd> + +**\<output_options\>** + +<dt>-n, -\\\-tail \<integer\> </dt> +<dd>Limits the output to the last \<integer\> of qualifying log entries found.</dd> + +<dt>-s, -\\\-slice \<offset\> \[\<limit\>\] </dt> +<dd>From the list of qualifying log entries, returns the \<limit\> number of entries starting at the \<offset\> entry number, where an \<offset\> of zero (`0`) denotes the first entry in the result set and an \<offset\> of any number greater than zero counts back from the end of the result set.</dd> + +<dt>-o, -\\\-out \<output\_file\> </dt> +<dd>Writes the output to the specified file or directory location instead of `STDOUT`.</dd> + +<dt>-z, -\\\-zip \<0..9\> </dt> +<dd>Compresses the output file to the specified compression level using `gzip`, where `0` is no compression and `9` is maximum compression. If you supply an output file name ending in `.gz`, the output file will be compressed by default using maximum compression.</dd> + +<dt>-a, -\\\-append </dt> +<dd>If the output file already exists, appends to the file instead of overwriting it.</dd> + + +**\<pattern\_matching\_options\>** + +<dt>-c, -\\\-case i\[gnore\] | r\[espect\] </dt> +<dd>Matching of alphabetic characters is case sensitive by default unless proceeded by the `--case=ignore` option.</dd> + +<dt>-C, -\\\-columns '\<string\>' </dt> +<dd>Selects specific columns from the log file. Specify the desired columns as a comma-delimited string of column numbers beginning with 1, where the second column from left is 2, the third is 3, and so on.</dd> + +<dt>-f, -\\\-find '\<string\>' </dt> +<dd>Finds the log entries containing the specified string.</dd> + +<dt>-F, -\\\-nofind '\<string\>' </dt> +<dd>Rejects the log entries containing the specified string.</dd> + +<dt>-m, -\\\-match \<regex\> </dt> +<dd>Finds log entries that match the specified Python regular expression. See [http://docs.python.org/library/re.html](http://docs.python.org/library/re.html) for Python regular expression syntax.</dd> + +<dt>-M, -\\\-nomatch \<regex\> </dt> +<dd>Rejects log entries that match the specified Python regular expression. See [http://docs.python.org/library/re.html](http://docs.python.org/library/re.html) for Python regular expression syntax.</dd> + +<dt>-t, -\\\-trouble </dt> +<dd>Finds only the log entries that have `ERROR:`, `FATAL:`, or `PANIC:` in the first line.</dd> + +**\<timestamp_options\>** + +<dt>-b, -\\\-begin \<datetime\> </dt> +<dd>Specifies a starting date and time to begin searching for log entries in the format of `YYYY-MM-DD [hh:mm[:ss]]`. + +If a time is specified, the date and time must be enclosed in either single or double quotes. This example encloses the date and time in single quotes: + +``` shell +$ gplogfilter -b '2016-02-13 14:23' +``` +</dd> + +<dt>-e, -\\\-end \<datetime\> </dt> +<dd>Specifies an ending date and time to stop searching for log entries in the format of `YYYY-MM-DD [hh:mm[:ss]]`. + +If a time is specified, the date and time must be enclosed in either single or double quotes. This example encloses the date and time in single quotes: + +``` shell +$ gplogfilter -e '2016-02-13 14:23' +``` +</dd> + +<dt>-d, -\\\-duration \<time\> </dt> +<dd>Specifies a time duration to search for log entries in the format of `[hh][:mm[:ss]]`. If used without either the `-b` or `-e` option, will use the current time as a basis.</dd> + +**Other Options** + +<dt>-\\\-help </dt> +<dd>Displays the online help.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + +## <a id="topic1__section9"></a>Examples + +Display the last three error messages in the identified log file: + +``` shell +$ gplogfilter -t -n 3 "/data/hawq/master/pg_log/hawq-2016-09-01_134934.csv" +``` + +Display the last five error messages in a date-specified log file: + +``` shell +$ gplogfilter -t -n 5 "/data/hawq-file-path/hawq-yyyy-mm-dd*.csv" +``` + +Display all log messages in the date-specified log file timestamped in the last 10 minutes: + +``` shell +$ gplogfilter -d :10 "/data/hawq-file-path/hawq-yyyy-mm-dd*.csv" +``` + +Display log messages in the identified log file containing the string `|con6 cmd11|`: + +``` shell +$ gplogfilter -f '|con6 cmd11|' "/data/hawq/master/pg_log/hawq-2016-09-01_134934.csv" +``` + +Using [hawq ssh](hawqssh.html#topic1), run `gplogfilter` on the segment hosts and search for log messages in the segment log files containing the string `con6` and save output to a file. + +``` shell +$ hawq ssh -f /data/hawq-2.x/segmentdd/pg_hba.conf -e 'source /usr/local/hawq/greenplum_path.sh ; +gplogfilter -f con6 /data/hawq-2.x/pg_log/hawq*.csv' > seglog.out +``` + +## <a id="topic1__section10"></a>See Also + +[hawq ssh](hawqssh.html#topic1) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/hawqactivate.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/hawqactivate.html.md.erb b/markdown/reference/cli/admin_utilities/hawqactivate.html.md.erb new file mode 100644 index 0000000..7afd6c7 --- /dev/null +++ b/markdown/reference/cli/admin_utilities/hawqactivate.html.md.erb @@ -0,0 +1,87 @@ +--- +title: hawq activate +--- + +Activates a standby master host and makes it the active master for the HAWQ system. + +**Note:** If HAWQ was installed using Ambari, do not use `hawq activate` to activate a standby master host. The system catalogs could become unsynchronized if you mix Ambari and command line functions. For Ambari-managed HAWQ clusters, always use the Ambari administration interface to activate a standby master. For more information, see [Manging HAWQ Using Ambari](../../../admin/ambari-admin.html#topic1). + +## <a id="topic1__section2"></a>Synopsis + +``` pre +hawq activate standby + [-M (smart|fast|immediate) | --mode (smart|fast|immediate)] + [-t <time> | --timeout <time>] + [-l <logfile_directory> | --logdir <logfile_directory>] + [(-v | --verbose) | (-q | --quiet)] + [--ignore-bad-hosts] + +hawq activate [-h | --help] + +``` + +## <a id="topic1__section3"></a>Description + +If the primary master fails, the log replication process is shut down, and the standby master can be activated in its place. The `hawq activate standby` utility activates a backup, standby master host and brings it into operation as the active master instance for a HAWQ system. The activated standby master effectively becomes the HAWQ master, accepting client connections on the master port. + +When you initialize a standby master, the default is to use the same port as the active master. For information about the master port for the standby master, see [hawq init](hawqinit.html#topic1). + +You must run this utility from the master host you are activating, not the failed master host you are disabling. Running this utility assumes you have a standby master host configured for the system . + +The utility will perform the following steps: + +- Stops the synchronization process (`walreceiver`) on the standby master +- Updates the system catalog tables of the standby master using the logs +- Activates the standby master to be the new active master for the system +- Restarts the HAWQ system with the new master host + +In order to use `hawq activate standby` to activate a new primary master host, the master host that was previously serving as the primary master cannot be running. The utility checks for a `postmaster.pid` file in the data directory of the disabled master host, and if it finds it there, it will assume the old master host is still active. In some cases, you may need to remove the `postmaster.pid` file from the disabled master host data directory before running `hawq activate standby` (for example, if the disabled master host process was terminated unexpectedly). + +After activating a standby master, run `ANALYZE` to update the database query statistics. For example: + +``` shell +$ psql <dbname> -c 'ANALYZE;' +``` + +After you activate the standby master as the primary master, the HAWQ system no longer has a standby master configured. You might want to specify another host to be the new standby with the [hawq init](hawqinit.html#topic1) utility. + +## <a id="topic1__section4"></a>Options + +<dt>-M, -\\\-mode (smart | fast | immediate) </dt> +<dd>Stop with one of the specified modes. + +Smart shutdown is the default. Shutdown fails with a warning message, if active connections are found. + +Fast shut down interrupts and rolls back any transactions currently in progress. + +Immediate shutdown aborts transactions in progress and kills all `postgres` processes without allowing the database server to complete transaction processing or clean up any temporary or in-process work files. Because of this, immediate shutdown is not recommended. In some instances, it can cause database corruption that requires manual recovery.</dd> + +<dt>-t, -\\\-timeout \<timeout\_seconds\> </dt> +<dd>Seconds to wait before discontinuing the operation. If not specified, the default timeout is 60 seconds.</dd> + +<dt>-l, -\\\-logdir \<logfile\_directory\> </dt> +<dd>Specifies the log directory for logs of the management tools. The default is `~/hawq/Adminlogs/`.</dd> + +<dt>-v, -\\\-verbose </dt> +<dd>Displays detailed status, progress and error messages output by the utility.</dd> + +<dt>-q, -\\\-quiet </dt> +<dd>Run in quiet mode. Command output is not displayed on the screen, but is still written to the log file.</dd> + +<dt>-\\\-ignore-bad-hosts </dt> +<dd>Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable.</dd> + +<dt>-h, -\\\-help (help) </dt> +<dd>Displays the online help.</dd> + +## <a id="topic1__section5"></a>Example + +Activate the standby master host and make it the active master instance for a HAWQ system (run from backup master host you are activating): + +``` shell +$ hawq activate standby +``` + +## <a id="topic1__section6"></a>See Also + +[hawq init](hawqinit.html#topic1) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/hawqcheck.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/hawqcheck.html.md.erb b/markdown/reference/cli/admin_utilities/hawqcheck.html.md.erb new file mode 100644 index 0000000..23a496d --- /dev/null +++ b/markdown/reference/cli/admin_utilities/hawqcheck.html.md.erb @@ -0,0 +1,126 @@ +--- +title: hawq check +--- + +Verifies and validates HAWQ platform settings. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +hawq check -f <hostfile_hawq_check> | (-h <hostname> | --host <hostname>) + [--hadoop <hadoop_home> | --hadoop-home <hadoop_home>] + [--config <config_file>] + [--stdout | --zipout] + [--kerberos] + [--hdfs-ha] + [--yarn] + [--yarn-ha] + +hawq check --zipin <hawq_check_zipfile> + +hawq check --version + +hawq check -? +``` + +## <a id="topic1__section3"></a>Description + +The `hawq check` utility determines the platform on which you are running HAWQ and validates various platform-specific configuration settings as well as HAWQ and HDFS-specific configuration settings. In order to perform HAWQ configuration checks, make sure HAWQ has been already started and `hawq config` works. For HDFS checks, you should either set the `$HADOOP_HOME` environment variable or provide the full path to the hadoop installation location using the `--hadoop` option. + +The `hawq check` utility can use a host file or a file previously created with the `--zipout `option to validate platform settings. If `GPCHECK_ERROR` displays, one or more validation checks failed. You can also use `hawq check` to gather and view platform settings on hosts without running validation checks. When running checks, `hawq check` compares your actual configuration setting with an expected value listed in a config file (`$GPHOME/etc/hawq_check.cnf` by default). You must modify your configuration values for "mount.points" and "diskusage.monitor.mounts" to reflect the actual mount points you want to check, as a comma-separated list. Otherwise, the utility only checks the root directory, which may not be helpful. + +An example is shown below: + +``` pre +[linux.mount] +mount.points = /,/data1,/data2 + +[linux.diskusage] +diskusage.monitor.mounts = /,/data1,/data2 +``` +## <a id="args"></a>Arguments + +<dt>-f \<hostfile\_hawq\_check\> </dt> +<dd>The name of a file that contains a list of hosts that `hawq check` uses to validate platform-specific settings. This file should contain a single host name for all hosts in your HAWQ system (master, standby master, and segments).</dd> + +<dt>-h, -\\\-host \<hostname\> </dt> +<dd>Specifies a single host on which platform-specific settings will be validated.</dd> + +<dt>-\\\-zipin \<hawq\_check\_zipfile\> </dt> +<dd>Use this option to decompress and check a .zip file created with the `--zipout` option. If you specify the `--zipin` option, `hawq check` performs validation tasks against the specified file.</dd> + + +## <a id="topic1__section4"></a>Options + + +<dt>-\\\-config \<config\_file\> </dt> +<dd>The name of a configuration file to use instead of the default file `$GPHOME/etc/hawq_check.cnf`.</dd> + +<dt>-\\\-hadoop, -\\\-hadoop-home \<hadoop\_home\> </dt> +<dd>Use this option to specify the full path to your hadoop installation location so that `hawq check` can validate HDFS settings. This option is not needed if the `$HADOOP_HOME` environment variable is set.</dd> + +<dt>-\\\-stdout </dt> +<dd>Send collected host information from `hawq check` to standard output. No checks or validations are performed.</dd> + +<dt>-\\\-zipout </dt> +<dd>Save all collected data to a .zip file in the current working directory. `hawq check` automatically creates the .zip file and names it `hawq_check_timestamp.tar.gz.` No checks or validations are performed.</dd> + +<dt>-\\\-kerberos </dt> +<dd>Use this option to check HDFS and YARN when running Kerberos mode. This allows `hawq check` to validate HAWQ/HDFS/YARN settings with Kerberos enabled.</dd> + +<dt>-\\\-hdfs-ha </dt> +<dd>Use this option to indicate that HDFS-HA mode is enabled, allowing `hawq check` to validate HDFS settings with HA mode enabled.</dd> + +<dt>-\\\-yarn </dt> +<dd>If HAWQ is using YARN, enables yarn mode, allowing `hawq check` to validate the basic YARN settings.</dd> + +<dt>-\\\-yarn-ha </dt> +<dd>Use this option to indicate HAWQ is using YARN with High Availability mode enabled, to allow `hawq check` to validate HAWQ-YARN settings with YARN-HA enabled.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + +<dt>-? (help) </dt> +<dd>Displays the online help.</dd> + +## <a id="topic1__section5"></a>Examples + +Verify and validate the HAWQ platform settings by entering a host file and specifying the full hadoop install path: + +``` shell +$ hawq check -f hostfile_hawq_check --hadoop /usr/hdp/<version>/hadoop +``` + +Verify and validate the HAWQ platform settings with HDFS HA enabled, YARN HA enabled and Kerberos enabled: + +``` shell +$ hawq check -f hostfile_hawq_check --hadoop /usr/local/hadoop-<version> --hdfs-ha --yarn-ha --kerberos +``` + +Verify and validate the HAWQ platform settings with HDFS HA enabled, and Kerberos enabled: + +``` shell +$ hawq check -f hostfile_hawq_check --hadoop /usr/hdp/<version>/hadoop --hdfs-ha --kerberos +``` + +Save HAWQ platform settings to a zip file, when the `$HADOOP_HOME` environment variable is set: + +``` shell +$ hawq check -f hostfile_hawq_check --zipout +``` + +Verify and validate the HAWQ platform settings using a zip file created with the `--zipout` option: + +``` shell +$ hawq check --zipin hawq_check_timestamp.tar.gz +``` + +View collected HAWQ platform settings: + +``` shell +$ hawq check -f hostfile_hawq_check --hadoop /usr/local/hadoop-<version> --stdout +``` + +## <a id="topic1__section6"></a>See Also + +[hawq checkperf](hawqcheckperf.html#topic1) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/hawqcheckperf.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/hawqcheckperf.html.md.erb b/markdown/reference/cli/admin_utilities/hawqcheckperf.html.md.erb new file mode 100644 index 0000000..f5c7e2c --- /dev/null +++ b/markdown/reference/cli/admin_utilities/hawqcheckperf.html.md.erb @@ -0,0 +1,137 @@ +--- +title: hawq checkperf +--- + +Verifies the baseline hardware performance of the specified hosts. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +hawq checkperf -d <test_directory> [-d <test_directory> ...] +    (-f <hostfile_checkperf> | - h <hostname> [-h <hostname> ...]) +    [-r ds] +    [-B <block_size>] +    [-S <file_size>] +    [-D] +    [-v|-V] + +hawq checkperf -d <temp_directory> +    (-f <hostfile_checknet> | - h <hostname> [-h <hostname> ...]) +    [-r n|N|M [--duration <time>] [--netperf]] +    [-D] +    [-v|-V] + +hawq checkperf --version + +hawq checkperf -? +``` + +## <a id="topic1__section3"></a>Description + +The `hawq checkperf` utility starts a session on the specified hosts and runs the following performance tests: + +- **Disk I/O Test (dd test)** â To test the sequential throughput performance of a logical disk or file system, the utility uses the **dd** command, which is a standard UNIX utility. It times how long it takes to write and read a large file to and from disk and calculates your disk I/O performance in megabytes (MB) per second. By default, the file size that is used for the test is calculated at two times the total random access memory (RAM) on the host. This ensures that the test is truly testing disk I/O and not using the memory cache. +- **Memory Bandwidth Test (stream)** â To test memory bandwidth, the utility uses the STREAM benchmark program to measure sustainable memory bandwidth (in MB/s). This tests that your system is not limited in performance by the memory bandwidth of the system in relation to the computational performance of the CPU. In applications where the data set is large (as in HAWQ), low memory bandwidth is a major performance issue. If memory bandwidth is significantly lower than the theoretical bandwidth of the CPU, then it can cause the CPU to spend significant amounts of time waiting for data to arrive from system memory. +- **Network Performance Test (gpnetbench\*)** â To test network performance (and thereby the performance of the HAWQ interconnect), the utility runs a network benchmark program that transfers a 5 second stream of data from the current host to each remote host included in the test. The data is transferred in parallel to each remote host and the minimum, maximum, average and median network transfer rates are reported in megabytes (MB) per second. If the summary transfer rate is slower than expected (less than 100 MB/s), you can run the network test serially using the `-r n` option to obtain per-host results. To run a full-matrix bandwidth test, you can specify `-r M` which will cause every host to send and receive data from every other host specified. This test is best used to validate if the switch fabric can tolerate a full-matrix workload. + +To specify the hosts to test, use the `-f` option to specify a file containing a list of host names, or use the `-h` option to name single host names on the command-line. If running the network performance test, all entries in the host file must be for network interfaces within the same subnet. If your segment hosts have multiple network interfaces configured on different subnets, run the network test once for each subnet. + +You must also specify at least one test directory (with `-d`). The user who runs `hawq checkperf` must have write access to the specified test directories on all remote hosts. For the disk I/O test, the test directories should correspond to your segment data directories. For the memory bandwidth and network tests, a temporary directory is required for the test program files. + +Before using `hawq checkperf`, you must have a trusted host setup between the hosts involved in the performance test. You can use the utility `hawq ssh-exkeys` to update the known host files and exchange public keys between hosts if you have not done so already. Note that `hawq checkperf` calls to `hawq ssh` and `hawq scp`, so these HAWQ utilities must also be in your `$PATH`. + +## <a id="args"></a>Arguments + +<dt>-d \<test\_directory\> </dt> +<dd>For the disk I/O test, specifies the file system directory locations to test. You must have write access to the test directory on all hosts involved in the performance test. You can use the `-d` option multiple times to specify multiple test directories (for example, to test disk I/O of your data directories).</dd> + +<dt>-d \<temp\_directory\> </dt> +<dd>For the network and stream tests, specifies a single directory where the test program files will be copied for the duration of the test. You must have write access to this directory on all hosts involved in the test.</dd> + +<dt>-f \<hostfile\_checkperf\> </dt> +<dd>For the disk I/O and stream tests, specifies the name of a file that contains one host name per host that will participate in the performance test. The host name is required, and you can optionally specify an alternate user name and/or SSH port number per host. The syntax of the host file is one host per line as follows: + +``` pre +[username@]hostname[:ssh_port] +``` +</dd> + +<dt>-f \<hostfile\_checknet\> </dt> +<dd>For the network performance test, all entries in the host file must be for host adresses within the same subnet. If your segment hosts have multiple network interfaces configured on different subnets, run the network test once for each subnet. For example (a host file containing segment host address names for interconnect subnet 1): + +``` pre +sdw1-1 +sdw2-1 +sdw3-1 +``` +</dd> + +<dt>-h \<hostname\> </dt> +<dd>Specifies a single host name (or host address) that will participate in the performance test. You can use the `-h` option multiple times to specify multiple host names.</dd> + +## <a id="topic1__section4"></a>Options + +<dt>-r ds{n|N|M} </dt> +<dd>Specifies which performance tests to run. The default is `dsn`: + +- Disk I/O test (`d`) +- Stream test (`s`) +- Network performance test in sequential (`n`), parallel (`N`), or full-matrix (`M`) mode. The optional `--duration` option specifies how long (in seconds) to run the network test. To use the parallel (`N`) mode, you must run the test on an *even* number of hosts. + + If you would rather use `netperf` ([http://www.netperf.org](http://www.netperf.org)) instead of the HAWQ network test, you can download it and install it into `$GPHOME/bin/lib` on all HAWQ hosts (master and segments). You would then specify the optional `--netperf` option to use the `netperf` binary instead of the default `gpnetbench*` utilities.</dd> + +<dt>-B \<block\_size\> </dt> +<dd>Specifies the block size (in KB or MB) to use for disk I/O test. The default is 32KB, which is the same as the HAWQ page size. The maximum block size is 1 MB.</dd> + +<dt>-S \<file\_size\> </dt> +<dd>Specifies the total file size to be used for the disk I/O test for all directories specified with `-d`. \<file\_size\> should equal two times total RAM on the host. If not specified, the default is calculated at two times the total RAM on the host where `hawq checkperf` is executed. This ensures that the test is truly testing disk I/O and not using the memory cache. You can specify sizing in KB, MB, or GB.</dd> + +<dt>-D (display per-host results) </dt> +<dd>Reports performance results for each host for the disk I/O tests. The default is to report results for just the hosts with the minimum and maximum performance, as well as the total and average performance of all hosts.</dd> + +<dt>-\\\-duration \<time\> </dt> +<dd>Specifies the duration of the network test in seconds (s), minutes (m), hours (h), or days (d). The default is 15 seconds.</dd> + +<dt>-\\\-netperf </dt> +<dd>Specifies that the `netperf` binary should be used to perform the network test instead of the HAWQ network test. To use this option, you must download `netperf` from [http://www.netperf.org](http://www.netperf.org) and install it into `$GPHOME/bin/lib` on all HAWQ hosts (master and segments).</dd> + +<dt>-v (verbose) | -V (very verbose) </dt> +<dd>Verbose mode shows progress and status messages of the performance tests as they are run. Very verbose mode shows all output messages generated by this utility.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + +<dt>-? (help) </dt> +<dd>Displays the online help.</dd> + +## <a id="topic1__section5"></a>Examples + +Run the disk I/O and memory bandwidth tests on all the hosts in the file *host\_file* using the test directory of */data1* and */data2*: + +``` shell +$ hawq checkperf -f hostfile_checkperf -d /data1 -d /data2 -r ds +``` + +Run only the disk I/O test on the hosts named *sdw1* and sdw2 using the test directory of */data1*. Show individual host results and run in verbose mode: + +``` shell +$ hawq checkperf -h sdw1 -h sdw2 -d /data1 -r d -D -v +``` + +Run the parallel network test using the test directory of */tmp,* where *hostfile\_check\_ic\** specifies all network interface host address names within the same interconnect subnet: + +``` shell +$ hawq checkperf -f hostfile_checknet_ic1 -r N -d /tmp +$ hawq checkperf -f hostfile_checknet_ic2 -r N -d /tmp +``` + +Run the same test as above, but use `netperf` instead of the HAWQ network test (note that `netperf` must be installed in `$GPHOME/bin/lib` on all HAWQ hosts): + +``` shell +$ hawq checkperf -f hostfile_checknet_ic1 -r N --netperf -d /tmp +$ hawq checkperf -f hostfile_checknet_ic2 -r N --netperf -d /tmp +``` + +## <a id="topic1__section6"></a>See Also + +[hawq ssh](hawqssh.html#topic1), [hawq scp](hawqscp.html#topic1) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/hawqconfig.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/hawqconfig.html.md.erb b/markdown/reference/cli/admin_utilities/hawqconfig.html.md.erb new file mode 100644 index 0000000..9f5e840 --- /dev/null +++ b/markdown/reference/cli/admin_utilities/hawqconfig.html.md.erb @@ -0,0 +1,134 @@ +--- +title: hawq config +--- + +Sets server configuration parameters on all nodes (master and segments) for HAWQ systems that are managed using command-line utilities. + +**Note:** If you install and manage HAWQ using Ambari, do not use `hawq config` to configure HAWQ properties. Ambari will overwrite any changes that were made by `hawq config` when it restarts the cluster. For Ambari-managed HAWQ clusters, always use the Ambari administration interface to set or change HAWQ configuration properties. + +## <a id="topic1__section2"></a>Synopsis + +``` pre +hawq config -c <hawq_property> | --change <hawq_property> + -v <hawq_property_value> | --value <hawq_property_value> + [--skipvalidation] [--ignore-bad-hosts] +       +hawq config -r <hawq_property> | --remove <hawq_property> + [--skipvalidation] [--ignore-bad-hosts] + +hawq config -l | --list + [--ignore-bad-hosts] + +hawq config -s <hawq_property> | --show <hawq_property> + [--ignore-bad-hosts] + +hawq config --help +``` + +## <a id="topic1__section3"></a>Description + +The `hawq config` utility allows you to set, unset, or view configuration properties from the `hawq-site.xml` files of all instances in your HAWQ system. + +**Note:** The `hawq config` utility makes configuration properties identical and consistent across all nodes, including the master and segments. Using the utility will override any unique configurations that were defined manually in `hawq-site.xml`. + +`hawq config` can only be used to manage specific properties. For example, you cannot use it to set properties such as `port`, which is required to be distinct for every segment instance. Use the `-l` (list) option to see a complete list of configuration properties supported by `hawq config`. + +When `hawq config` sets a configuration property in a `hawq_site.xml` file, the new property setting always displays at the bottom of the file. When you use `hawq config` to remove a configuration property setting, `hawq config` comments out the property in all `hawq-site.xml` files, thereby restoring the system default setting. For example, if you use `hawq config `to remove (comment out) a property and later add it back (set a new value), there will be two instances of the property; one that is commented out, and one that is enabled and inserted at the bottom of the `hawq-site.xml` file. + +After setting a property, you must restart your HAWQ system or reload the `hawq-site.xml` file for the change to take effect. Whether you require a restart or a reload depends on the property being set. To reload the configuration files, use `hawq stop cluster -u`. To restart the system, use `hawq restart` . + +To show the currently set values for a property across the system, use the `-s` option. + +`hawq config` uses the following environment variables to connect to the HAWQ master instance and obtain system configuration information: + +- `PGHOST` +- `PGPORT` +- `PGUSER` +- `PGPASSWORD` +- `PGDATABASE` + +## <a id="topic1__section4"></a>Options + +<dt> +-c, -\\\-change \<hawq\_property\> +</dt> +<dd>Changes a HAWQ property setting by adding the new setting to the bottom of the `hawq-site.xml` files.</dd> + +<dt> +-v, -\\\-value \<hawq\_property\_value\> +</dt> +<dd> +Set the value of the HAWQ property setting in the `hawq-site.xml` files. +</dd> + +<dt> +-r, -\\\-remove \<hawq\_property\> +</dt> +<dd> +Removes a HAWQ property setting by commenting out the entry in the `hawq-site.xml` files. +</dd> + +<dt> +-s, -\\\-show \<hawq\_property\> +</dt> +<dd> +Shows the value for a HAWQ property name used on all instances (master and segments) in the HAWQ system. If there is a discrepancy in a parameter value between segment instances, the `hawq config` utility displays an error message. Note that the `hawq config` utility reads property values directly from the database, and not the `hawq-site.xml` file. If you are using `hawq config` to set properties across all segments, then running `hawq config -s` to verify the changes, you might still see the previous (old) values. You must reload the configuration files (`hawq stop cluster -u`) or restart the system (`hawq restart`) for changes to take effect. +</dd> + +<dt> +-l, -\\\-list +</dt> +<dd> +Lists all HAWQ property settings supported by the `hawq config` utility. +</dd> + +<dt> +-\\\-skipvalidation +</dt> +<dd> +Overrides the system validation checks of `hawq config` and allows you to operate on any server property, including hidden parameters and restricted parameters that cannot be changed by `hawq config`. Do not modify hidden or restricted parameters unless you are aware of all potential consequences. +</dd> + +<dt> +-\\\-ignore-bad-hosts +</dt> +<dd> +Overrides copying configuration files to a host on which SSH validation fails. If ssh to a skipped host is reestablished, make sure the configuration files are re-synched once it is reachable. +</dd> + +<dt> +-h, -\\\-help +</dt> +<dd> +Displays the online help. +</dd> + +## <a id="topic1__section5"></a>Examples + +Set the `max_connections` setting to 100: + +``` shell +$ hawq config -c max_connections -v 100 +``` + +Comment out all instances of the `default_statistics_target` property, and restore the system default: + +``` shell +$ hawq config -r default_statistics_target +``` + +List all properties supported by `hawq config`: + +``` shell +$ hawq config -l +``` + +Show the values of a particular property across the system: + +``` shell +$ hawq config -s max_connections +``` + +## <a id="topic1__section6"></a>See Also + +[hawq stop](hawqstop.html#topic1) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/reference/cli/admin_utilities/hawqextract.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/reference/cli/admin_utilities/hawqextract.html.md.erb b/markdown/reference/cli/admin_utilities/hawqextract.html.md.erb new file mode 100644 index 0000000..b338523 --- /dev/null +++ b/markdown/reference/cli/admin_utilities/hawqextract.html.md.erb @@ -0,0 +1,319 @@ +--- +title: hawq extract +--- + +Extracts the metadata of a specified table into a YAML file. + +## Synopsis + +``` pre +hawq extract [<connection_options>] [-o <output_file>] <tablename> + +hawq extract -? + +hawq extract --version +``` +where: + +``` pre +<connection_options> = + [-h <host>] + [-p <port>] + [-U <username>] + [-d <database>] + [-W] +``` + +## Description + +`hawq extract` is a utility that extracts a table's metadata into a YAML formatted file. HAWQ's InputFormat uses this YAML-formatted file to read a HAWQ file stored on HDFS directly into the MapReduce program. The YAML configuration file can also be used provide the metadata for registering files into HAWQ with the `hawq register` command. + +**Note:** +`hawq extract` is bound by the following rules: + +- You must start up HAWQ to use `hawq extract`. +- `hawq extract` only supports AO and Parquet tables. +- `hawq extract` supports partitioned tables, but does not support sub-partitions. + +## Arguments + +<dt><tablename> </dt> +<dd>Name of the table that you need to extract metadata. You can use the format *namespace\_name.table\_name*.</dd> + +## Options + +<dt>-o <output\_file> </dt> +<dd>Is the name of a file that `hawq extract` uses to write the metadata. If you do not specify a name, `hawq extract` writes to `stdout`.</dd> + +<dt>-v (verbose mode) </dt> +<dd>Displays the verbose output of the extraction process.</dd> + +<dt>-? (help) </dt> +<dd>Displays the online help.</dd> + +<dt>-\\\-version </dt> +<dd>Displays the version of this utility.</dd> + +**<connection_options>** + +<dt>-h <host> </dt> +<dd>Specifies the host name of the machine on which the HAWQ master database server is running. If not specified, it reads from the environment variable `$PGHOST` or defaults to `localhost`.</dd> + +<dt>-p <port> </dt> +<dd>Specifies the TCP port on which the HAWQ master database server is listening for connections. If not specified, reads from the environment variable `$PGPORT` or defaults to 5432.</dd> + +<dt>-U <username> </dt> +<dd>The database role name to connect as. If not specified, reads from the environment variable `$PGUSER` or defaults to the current system user name.</dd> + +<dt>-d <database> </dt> +<dd>The database to connect to. If not specified, it reads from the environment variable `$PGDATABASE` or defaults to `template1`.</dd> + +<dt>-W (force password prompt) </dt> +<dd>Force a password prompt. If not specified, reads the password from the environment variable `$PGPASSWORD` or from a password file specified by `$PGPASSFILE` or in `~/.pgpass`.</dd> + +## Metadata File Format + +`hawq extract` exports the table metadata into a file using YAML 1.1 document format. The file contains various key information about the table, such as table schema, data file locations and sizes, partition constraints and so on. + +The basic structure of the metadata file is as follows: + +``` pre +Version: string (1.0.0) +DBVersion: string +FileFormat: string (AO/Parquet) +TableName: string (schemaname.tablename) +DFS_URL: string (hdfs://127.0.0.1:9000) +Encoding: UTF8 +AO_Schema: + - name: string + type: string + Bucketnum: 6 + Distribution_policy: DISTRIBUTED RANDOMLY + +AO_FileLocations: + Blocksize: int + Checksum: boolean + CompressionType: string + CompressionLevel: int + PartitionBy: string ('PARTITION BY ...') + Files: + - path: string (/gpseg0/16385/35469/35470.1) + size: long + + Partitions: + - Blocksize: int + Checksum: Boolean + CompressionType: string + CompressionLevel: int + Name: string + Constraint: string (PARTITION Jan08 START (date '2008-01-01') INCLUSIVE) + Files: + - path: string + size: long + +Parquet_Schema: + - name: string + type: string +  +Parquet_FileLocations: + RowGroupSize: long + PageSize: long + CompressionType: string + CompressionLevel: int + Checksum: boolean + EnableDictionary: boolean + PartitionBy: string + Files: + - path: string + size: long + Partitions: + - Name: string + RowGroupSize: long + PageSize: long + CompressionType: string + CompressionLevel: int + Checksum: boolean + EnableDictionary: boolean + Constraint: string + Files: + - path: string + size: long +``` + +## Example - Extracting an AO table + +Extract the `rank` table's metadata into a file named `rank_table.yaml`: + +``` shell +$ hawq extract -o rank_table.yaml -d postgres rank +``` + +**Output content in rank\_table.yaml** + +``` pre +AO_FileLocations: + Blocksize: 32768 + Checksum: false + CompressionLevel: 0 + CompressionType: null + Files: + - path: /gpseg0/16385/35469/35692.1 + size: 0 + - path: /gpseg1/16385/35469/35692.1 + size: 0 + PartitionBy: PARTITION BY list (gender) + Partitions: + - Blocksize: 32768 + Checksum: false + CompressionLevel: 0 + CompressionType: null + Constraint: PARTITION girls VALUES('F') WITH (appendonly=true) + Files: + - path: /gpseg0/16385/35469/35697.1 + size: 0 + - path: /gpseg1/16385/35469/35697.1 + size: 0 + Name: girls + - Blocksize: 32768 + Checksum: false + CompressionLevel: 0 + CompressionType: null + Constraint: PARTITION boys VALUES('M') WITH (appendonly=true) + Files: + - path: /gpseg0/16385/35469/35703.1 + size: 0 + - path: /gpseg1/16385/35469/35703.1 + size: 0 + Name: boys + - Blocksize: 32768 + Checksum: false + CompressionLevel: 0 + CompressionType: null + Constraint: DEFAULT PARTITION other WITH appendonly=true) + Files: + - path: /gpseg0/16385/35469/35709.1 + size: 90071728 + - path: /gpseg1/16385/35469/35709.1 + size: 90071512 + Name: other + AO_Schema: + - name: id + type: int4 + - name: rank + type: int4 + - name: year + type: int4 + - name: gender + type: bpchar + - name: count + type: int4 + DFS_URL: hdfs://127.0.0.1:9000 + Distribution_policy: DISTRIBUTED RANDOMLY + Encoding: UTF8 + FileFormat: AO + TableName: public.rank + Version: 1.0.0 +``` + +## Example - Extracting a Parquet table + +Extract the `orders` table's metadata into a file named `orders.yaml`: + +``` shell +$ hawq extract -o orders.yaml -d postgres orders +``` + +**Output content in orders.yaml** + +``` pre +DFS_URL: hdfs://127.0.0.1:9000 +Encoding: UTF8 +FileFormat: Parquet +TableName: public.orders +Version: 1.0.0 +Parquet_FileLocations: + Checksum: false + CompressionLevel: 0 + CompressionType: none + EnableDictionary: false + Files: + - path: /hawq-data/gpseg0/16385/16626/16657.1 + size: 0 + - path: /hawq-data/gpseg1/16385/16626/16657.1 + size: 0 + PageSize: 1048576 + PartitionBy: PARTITION BY range (o_orderdate) + Partitions: + - Checksum: false + CompressionLevel: 0 + CompressionType: none + Constraint: PARTITION p1_1 START ('1992-01-01'::date) END ('1994-12-31'::date) + EVERY ('3 years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576, + rowgroupsize=8388608, compresstype=none, compresslevel=0) + EnableDictionary: false + Files: + - path: /hawq-data/gpseg0/16385/16626/16662.1 + size: 8140599 + - path: /hawq-data/gpseg1/16385/16626/16662.1 + size: 8099760 + Name: orders_1_prt_p1_1 + PageSize: 1048576 + RowGroupSize: 8388608 + - Checksum: false + CompressionLevel: 0 + CompressionType: none + Constraint: PARTITION p1_11 START ('1995-01-01'::date) END ('1997-12-31'::date) + EVERY ('e years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576, + rowgroupsize=8388608, compresstype=none, compresslevel=0) + EnableDictionary: false + Files: + - path: /hawq-data/gpseg0/16385/16626/16668.1 + size: 8088559 + - path: /hawq-data/gpseg1/16385/16626/16668.1 + size: 8075056 + Name: orders_1_prt_p1_11 + PageSize: 1048576 + RowGroupSize: 8388608 + - Checksum: false + CompressionLevel: 0 + CompressionType: none + Constraint: PARTITION p1_21 START ('1998-01-01'::date) END ('2000-12-31'::date) + EVERY ('3 years'::interval) WITH (appendonly=true, orientation=parquet, pagesize=1048576, + rowgroupsize=8388608, compresstype=none, compresslevel=0) + EnableDictionary: false + Files: + - path: /hawq-data/gpseg0/16385/16626/16674.1 + size: 8065770 + - path: /hawq-data/gpseg1/16385/16626/16674.1 + size: 8126669 + Name: orders_1_prt_p1_21 + PageSize: 1048576 + RowGroupSize: 8388608 + RowGroupSize: 8388608 + Parquet_Schema: + - name: o_orderkey + type: int8 + - name: o_custkey + type: int4 + - name: o_orderstatus + type: bpchar + - name: o_totalprice + type: numeric + - name: o_orderdate + type: date + - name: o_orderpriority + type: bpchar + - name: o_clerk + type: bpchar + - name: o_shippriority + type: int4 + - name: o_comment + type: varchar + Distribution_policy: DISTRIBUTED RANDOMLY +``` + +## See Also + +[hawq load](hawqload.html#topic1), [hawq register](hawqregister.html#topic1) + +
