http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/admin/ambari-admin.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/admin/ambari-admin.html.md.erb b/markdown/admin/ambari-admin.html.md.erb new file mode 100644 index 0000000..a5b2169 --- /dev/null +++ b/markdown/admin/ambari-admin.html.md.erb @@ -0,0 +1,439 @@ +--- +title: Managing HAWQ Using Ambari +--- + +Ambari provides an easy interface to perform some of the most common HAWQ and PXF Administration Tasks. + +## <a id="amb-yarn"></a>Integrating YARN for Resource Management + +HAWQ supports integration with YARN for global resource management. In a YARN managed environment, HAWQ can request resources (containers) dynamically from YARN, and return resources when HAWQâs workload is not heavy. + +See also [Integrating YARN with HAWQ](../resourcemgmt/YARNIntegration.html) for command-line instructions and additional details about using HAWQ with YARN. + +### When to Perform + +Follow this procedure if you have already installed YARN and HAWQ, but you are currently using the HAWQ Standalone mode (not YARN) for resource management. This procedure helps you configure YARN and HAWQ so that HAWQ uses YARN for resource management. This procedure assumes that you will use the default YARN queue for managing HAWQ. + +### Procedure + +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Select **HAWQ** from the list of installed services. +3. Select the **Configs** tab, then the **Settings** tab. +4. Use the **Resource Manager** menu to change select the **YARN** option. +5. Click **Save**.<br/><br/>HAWQ will use the default YARN queue, and Ambari automatically configures settings for `hawq_rm_yarn_address`, `hawq_rm_yarn_app_name`, and `hawq_rm_yarn_scheduler_address` in the `hawq-site.xml` file.<br/><br/>If YARN HA was enabled, Ambari also automatically configures the `yarn.resourcemanager.ha` and `yarn.resourcemanager.scheduler.ha` properties in `yarn-site.xml`. +6. If you are using HDP 2.3, follow these additional instructions: + 1. Select **YARN** from the list of installed services. + 2. Select the **Configs** tab, then the **Advanced** tab. + 3. Expand the **Advanced yarn-site** section. + 4. Locate the `yarn.resourcemanager.system-metrics-publisher.enabled` property and change its value to `false`. + 5. Click **Save**. +6. (Optional.) When HAWQ is integrated with YARN and has no workload, HAWQ does not acquire any resources right away. HAWQâs resource manager only requests resources from YARN when HAWQ receives its first query request. In order to guarantee optimal resource allocation for subsequent queries and to avoid frequent YARN resource negotiation, you can adjust `hawq_rm_min_resource_perseg` so HAWQ receives at least some number of YARN containers per segment regardless of the size of the initial query. The default value is 2, which means HAWQâs resource manager acquires at least 2 YARN containers for each segment even if the first queryâs resource request is small.<br/><br/>This configuration property cannot exceed the capacity of HAWQâs YARN queue. For example, if HAWQâs queue capacity in YARN is no more than 50% of the whole cluster, and each YARN node has a maximum of 64GB memory and 16 vcores, then `hawq_rm_min_resource_perseg` in HAWQ cannot be set to more than 8 since HAW Qâs resource manager acquires YARN containers by vcore. In the case above, the HAWQ resource manager acquires a YARN container quota of 4GB memory and 1 vcore.<br/><br/>To change this parameter, expand **Custom hawq-site** and click **Add Property ...** Then specify `hawq_rm_min_resource_perseg` as the key and enter the desired Value. Click **Add** to add the property definition. +7. (Optional.) If the level of HAWQâs workload is lowered, then HAWQ's resource manager may have some idle YARN resources. You can adjust `hawq_rm_resource_idle_timeout` to let the HAWQ resource manager return idle resources more quickly or more slowly.<br/><br/>For example, when HAWQ's resource manager has to reacquire resources, it can cause latency for query resource requests. To let HAWQ resource manager retain resources longer in anticipation of an upcoming workload, increase the value of `hawq_rm_resource_idle_timeout`. The default value of `hawq_rm_resource_idle_timeout` is 300 seconds.<br/><br/>To change this parameter, expand **Custom hawq-site** and click **Add Property ...** Then specify `hawq_rm_resource_idle_timeout` as the key and enter the desired Value. Click **Add** to add the property definition. +8. Click **Save** to save your configuration changes. + +## <a id="move_yarn_rm"></a>Moving a YARN Resource Manager + +If you are using YARN to manage HAWQ resources and need to move a YARN resource manager, then you must update your HAWQ configuration. + +### When to Perform + +Use one of the following procedures to move YARN resource manager component from one node to another when HAWQ is configured to use YARN as the global resource manager (`hawq_global_rm_type` is `yarn`). The exact procedure you should use depends on whether you have enabled high availability in YARN. + +**Note:** In a Kerberos-secured environment, you must update <code>hadoop.proxyuser.yarn.hosts</code> property in HDFS <code>core-site.xml</code> before running a service check. The values should be set to the current YARN Resource Managers.</p> + +### Procedure (Single YARN Resource Manager) + +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +1. Click **YARN** in the list of installed services. +1. Select **Move ResourceManager**, and complete the steps in the Ambari wizard to move the Resource Manager to a new host. +1. After moving the Resource Manager successfully in YARN, click **HAWQ** in the list of installed services. +1. On the HAWQ **Configs** page, select the **Advanced** tab. +1. Under Advanced hawq-site section, update the following HAWQ properties: + - `hawq_rm_yarn_address`. Enter the same value defined in the `yarn.resourcemanager.address` property of `yarn-site.xml`. + - `hawq_rm_yarn_scheduler_address`. Enter the same value in the `yarn.resourcemanager.scheduler.address` property of `yarn-site.xml`. +1. Restart all HAWQ components so that the configurations get updated on all HAWQ hosts. +1. Run HAWQ Service Check, as described in [Performing a HAWQ Service Check](#amb-service-check), to ensure that HAWQ is operating properly. + +### Procedure (Highly Available YARN Resource Managers) + +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +1. Click **YARN** in the list of installed services. +1. Select **Move ResourceManager**, and complete the steps in the Ambari wizard to move the Resource Manager to a new host. +1. After moving the Resource Manager successfully in YARN, click **HAWQ** in the list of installed services. +1. On the HAWQ **Configs** page, select the **Advanced** tab. +1. Under `Custom yarn-client` section, update the HAWQ properties `yarn.resourcemanager.ha` and `yarn.resourcemanager.scheduler.ha`. These parameter values should be updated to match the corresponding parameters for the YARN service. Check the values under **ResourceManager hosts** in the **Resource Manager** section of the **Advanced** configurations for the YARN service. +1. Restart all HAWQ components so that the configuration change is updated on all HAWQ hosts. You can ignore the warning about the values of `hawq_rm_yarn_address` and `hawq_rm_yarn_scheduler_address` in `hawq-site.xml` not matching the values in `yarn-site.xml`, and click **Proceed Anyway**. +1. Run HAWQ Service Check, as described in [Performing a HAWQ Service Check](#amb-service-check), to ensure that HAWQ is operating properly. + + +## <a id="amb-service-check"></a>Performing a HAWQ Service Check + +A HAWQ Service check uses the `hawq state` command to display the configuration and status of segment hosts in a HAWQ Cluster. It also performs tests to ensure that HAWQ can write to and read from tables, and to ensure that HAWQ can write to and read from HDFS external tables using PXF. + +### When to Perform +* Execute this procedure immediately after any common maintenance operations, such as adding, activating, or removing the HAWQ Master Standby. +* Execute this procedure as a first step in troubleshooting problems in accessing HDFS data. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +4. Select **Service Actions > Run Service Check**, then click **OK** to perform the service check. + + Ambari displays the **HAWQ Service Check** task in the list of background operations. If any test fails, then Ambari displays a red error icon next to the task. +5. Click the **HAWQ Service Check** task to view the actual log messages that are generated while performing the task. The log messages display the basic configuration and status of HAWQ segments, as well as the results of the HAWQ and PXF tests (if PXF is installed). + +6. Click **OK** to dismiss the log messages or list of background tasks. + +## <a id="amb-config-check"></a>Performing a Configuration Check + +A configuration check determines if operating system parameters on the HAWQ host machines match their recommended settings. You can also perform this procedure from the command line using the `hawq check` command. The `hawq check` command is run against all HAWQ hosts. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +3. (Optional) Perform this step if you want to view or modify the host configuration parameters that are evaluated during the HAWQ config check: + 1. Select the **Configs** tab, then select the **Advanced** tab in the settings. + 1. Expand **Advanced Hawq Check** to view or change the list of parameters that are checked with a `hawq check` command or with the Ambari HAWQ Config check. + + **Note:** All parameter entries are stored in the `/usr/local/hawq/etc/hawq_check.cnf` file. Click the **Set Recommended** button if you want to restore the file to its original contents. +4. Select **Service Actions > Run HAWQ Config Check**, then click **OK** to perform the configuration check. + + Ambari displays the **Run HAWQ Config Check** task in the list of background operations. If any parameter does not meet the specification defined in `/usr/local/hawq/etc/hawq_check.cnf`, then Ambari displays a red error icon next to the task. +5. Click the **Run HAWQ Config Check** task to view the actual log messages that are generated while performing the task. Address any configuration errors on the indicated host machines. + +6. Click **OK** to dismiss the log messages or list of background tasks. + +## <a id="amb-restart"></a>Performing a Rolling Restart +Ambari provides the ability to restart a HAWQ cluster by restarting one or more segments at a time until all segments (or all segments with stale configurations) restart. You can specify a delay between restarting segments, and Ambari can stop the process if a specified number of segments fail to restart. Performing a rolling restart in this manner can help ensure that some HAWQ segments are available to service client requests. + +**Note:** If you do not need to preserve client connections, you can instead perform an full restart of the entire HAWQ cluster using **Service Actions > Restart All**. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +3. Select **Service Actions > Restart HAWQ Segments**. +4. In the Restart HAWQ Segments page: + * Specify the number of segments that you want Ambari to restart at a time. + * Specify the number of seconds Ambari should wait before restarting the next batch of HAWQ segments. + * Specify the number of restart failures that may occur before Ambari stops the rolling restart process. + * Select **Only restart HAWQ Segments with stale configs** if you want to limit the restart process to those hosts. + * Select **Turn On Maintenance Mode for HAWQ** to enable maintenance mode before starting the rolling restart process. This suppresses alerts that are normally generated when a segment goes offline. +5. Click **Trigger Rolling Restart** to begin the restart process. + + Ambari displays the **Rolling Restart of HAWQ segments** task in the list of background operations, and indicates the current batch of segments that it is restarting. Click the name of the task to view the log messages generated during the restart. If any segment fails to restart, Ambari displays a red warning icon next to the task. + +## <a id="bulk-lifecycle"></a>Performing Host-Level Actions on HAWQ Segment and PXF Hosts + +Ambari host-level actions enable you to perform actions on one or more hosts in the cluster at once. With HAWQ clusters, you can apply the **Start**, **Stop**, or **Restart** actions to one or more HAWQ segment hosts or PXF hosts. Using the host-level actions saves you the trouble of accessing individual hosts in Ambari and applying service actions one-by-one. + +### When to Perform +* Use the Ambari host-level actions when you have a large number of hosts in your cluster and you want to start, stop, or restart all HAWQ segment hosts or all PXF hosts as part of regularly-scheduled maintenance. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Select the **Hosts** tab at the top of the screen to display a list of all hosts in the cluster. +3. To apply a host-level action to all HAWQ segment hosts or PXF hosts, select an action using the applicable menu: + * **Actions > Filtered Hosts > HAWQ Segments >** [ **Start** | **Stop** | **Restart** ] + * **Actions > Filtered Hosts > PXF Hosts >** [ **Start** | **Stop** | **Restart** ] +4. To apply a host level action to a subset of HAWQ segments or PXF hosts: + 1. Filter the list of available hosts using one of the filter options: + * **Filter > HAWQ Segments** + * **Filter > PXF Hosts** + 2. Use the check boxes to select the hosts to which you want to apply the action. + 3. Select **Actions > Selected Hosts >** [ **Start** | **Stop** | **Restart** ] to apply the action to your selected hosts. + + +## <a id="amb-expand"></a>Expanding the HAWQ Cluster + +Apache HAWQ supports dynamic node expansion. You can add segment nodes while HAWQ is running without having to suspend or terminate cluster operations. + +### Guidelines for Cluster Expansion + +This topic provides some guidelines around expanding your HAWQ cluster. + +There are several recommendations to keep in mind when modifying the size of your running HAWQ cluster: + +- When you add a new node, install both a DataNode and a HAWQ segment on the new node. If you are using YARN to manage HAWQ resources, you must also configure a YARN NodeManager on the new node. +- After adding a new node, you should always rebalance HDFS data to maintain cluster performance. +- Adding or removing a node also necessitates an update to the HDFS metadata cache. This update will happen eventually, but can take some time. To speed the update of the metadata cache, select the **Service Actions > Clear HAWQ's HDFS Metadata Cache** option in Ambari. +- Note that for hash distributed tables, expanding the cluster will not immediately improve performance since hash distributed tables use a fixed number of virtual segments. In order to obtain better performance with hash distributed tables, you must redistribute the table to the updated cluster by either the [ALTER TABLE](../reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](../reference/sql/CREATE-TABLE-AS.html) command. +- If you are using hash tables, consider updating the `default_hash_table_bucket_number` server configuration parameter to a larger value after expanding the cluster but before redistributing the hash tables. + +### Procedure +First ensure that the new node(s) has been configured per the instructions found in [Apache HAWQ System Requirements](../requirements/system-requirements.html) and [Select HAWQ Host Machines](../install/select-hosts.html). + +1. If you have any user-defined function (UDF) libraries installed in your existing HAWQ cluster, install them on the new node(s) that you want to add to the HAWQ cluster. +2. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +3. Click **HAWQ** in the list of installed services. +4. Select the **Configs** tab, then select the **Advanced** tab in the settings. +5. Expand the **General** section, and ensure that the **Exchange SSH Keys** property (`hawq_ssh_keys`) is set to `true`. Change this property to `true` if needed, and click **Save** to continue. Ambari must be able to exchange SSH keys with any hosts that you add to the cluster in the following steps. +6. Select the **Hosts** tab at the top of the screen to display the Hosts summary. +7. If the host(s) that you want to add are not currently listed in the Hosts summary page, follow these steps: + 1. Select **Actions > Add New Hosts** to start the Add Host Wizard. + 2. Follow the initial steps of the Add Host Wizard to identify the new host, specify SSH keys or manually register the host, and confirm the new host(s) to add. + + See [Set Up Password-less SSH](http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Installing_HDP_AMB/content/_set_up_password-less_ssh.html) in the HDP documentation if you need more information about performing these tasks. + 3. When you reach the Assign Slaves and Clients page, ensure that the **DataNode**, **HAWQ Segment**, and **PXF** (if the PXF service is installed) components are selected. Select additional components as necessary for your cluster. + 4. Complete the wizard to add the new host and install the selected components. +8. If the host(s) that you want to add already appear in the Hosts summary, follow these steps: + 1. Click the hostname that you want to add to the HAWQ cluster from the list of hosts. + 2. In the Components summary, ensure that the host already runs the DataNode component. If it does not, select **Add > DataNode** and then click **Confirm Add**. Click **OK** when the task completes. + 3. In the Components summary, select **Add > HAWQ Segment**. + 4. Click **Confirm Add** to acknowledge the component to add. Click **OK** when the task completes. + 5. In the Components summary, select **Add > PXF**. + 6. Click **Confirm Add** to acknowledge the component to add. Click **OK** when the task completes. +17. (Optional) If you are using hash tables, adjust the **Default buckets for Hash Distributed tables** setting (`default_hash_table_bucket_number`) on the HAWQ service's **Configs > Settings** tab. Update this property's value by multiplying the new number of nodes in the cluster by the appropriate number indicated below. + + |Number of Nodes After Expansion|Suggested default\_hash\_table\_bucket\_number value| + |---------------|------------------------------------------| + |<= 85|6 \* \#nodes| + |\> 85 and <= 102|5 \* \#nodes| + |\> 102 and <= 128|4 \* \#nodes| + |\> 128 and <= 170|3 \* \#nodes| + |\> 170 and <= 256|2 \* \#nodes| + |\> 256 and <= 512|1 \* \#nodes| + |\> 512|512| +18. Ambari requires the HAWQ service to be restarted in order to apply the configuration changes. If you need to apply the configuration *without* restarting HAWQ (for dynamic cluster expansion), then you can use the HAWQ CLI commands described in [Manually Updating the HAWQ Configuration](#manual-config-steps) *instead* of following this step. + <br/><br/>Stop and then start the HAWQ service to apply your configuration changes via Ambari. Select **Service Actions > Stop**, followed by **Service Actions > Start** to ensure that the HAWQ Master starts before the newly-added segment. During the HAWQ startup, Ambari exchanges ssh keys for the `gpadmin` user, and applies the new configuration. + >**Note:** Do not use the **Restart All** service action to complete this step. +19. Consider the impact of rebalancing HDFS to other components, such as HBase, before you complete this step. + <br/><br/>Rebalance your HDFS data by selecting the **HDFS** service and then choosing **Service Actions > Rebalance HDFS**. Follow the Ambari instructions to complete the rebalance action. +20. Speed up the clearing of the metadata cache by first selecting the **HAWQ** service and then selecting **Service Actions > Clear HAWQ's HDFS Metadata Cache**. +21. If you are using hash distributed tables and wish to take advantage of the performance benefits of using a larger cluster, redistribute the data in all hash-distributed tables by using either the [ALTER TABLE](../reference/sql/ALTER-TABLE.html) or [CREATE TABLE AS](../reference/sql/CREATE-TABLE-AS.html) command. You should redistribute the table data if you modified the `default_hash_table_bucket_number` configuration parameter. + + **Note:** The redistribution of table data can take a significant amount of time. +22. (Optional.) If you changed the **Exchange SSH Keys** property value before adding the host(s), change the value back to `false` after Ambari exchanges keys with the new hosts. This prevents Ambari from exchanging keys with all hosts every time the HAWQ master is started or restarted. + +23. (Optional.) If you enabled temporary password-based authentication while preparing/configuring your HAWQ host systems, turn off password-based authentication as described in [Apache HAWQ System Requirements](../requirements/system-requirements.html#topic_pwdlessssh). + +#### <a id="manual-config-steps"></a>Manually Updating the HAWQ Configuration +If you need to expand your HAWQ cluster without restarting the HAWQ service, follow these steps to manually apply the new HAWQ configuration. (Use these steps *instead* of following Step 7 in the above procedure.): + +1. Update your configuration to use the new `default_hash_table_bucket_number` value that you calculated: + 1. SSH into the HAWQ master host as the `gpadmin` user: + ```shell + $ ssh gpadmin@<HAWQ_MASTER_HOST> + ``` + 2. Source the `greenplum_path.sh` file to update the shell environment: + ```shell + $ source /usr/local/hawq/greenplum_path.sh + ``` + 3. Verify the current value of `default_hash_table_bucket_number`: + ```shell + $ hawq config -s default_hash_table_bucket_number + ``` + 4. Update `default_hash_table_bucket_number` to the new value that you calculated: + ```shell + $ hawq config -c default_hash_table_bucket_number -v <new_value> + ``` + 5. Reload the configuration without restarting the cluster: + ```shell + $ hawq stop cluster -u + ``` + 6. Verify that the `default_hash_table_bucket_number` value was updated: + ```shell + $ hawq config -s default_hash_table_bucket_number + ``` +2. Edit the `/usr/local/hawq/etc/slaves` file and add the new HAWQ hostname(s) to the end of the file. Separate multiple hosts with new lines. For example, after adding host4 and host5 to a cluster already contains hosts 1-3, the updated file contents would be: + + ``` + host1 + host2 + host3 + host4 + host5 + ``` +3. Continue with Step 8 in the previous procedure, [Expanding the HAWQ Cluster](#amb-expand). When the HAWQ service is ready to be restarted via Ambari, Ambari will refresh the new configurations. + +## <a id="amb-activate-standby"></a>Activating the HAWQ Standby Master +Activating the HAWQ Standby Master promotes the standby host as the new HAWQ Master host. The previous HAWQ Master configuration is automatically removed from the cluster. + +### When to Perform +* Execute this procedure immediately if the HAWQ Master fails or becomes unreachable. +* If you want to take the current HAWQ Master host offline for maintenance, execute this procedure during a scheduled maintenance period. This procedure requires a restart of the HAWQ service. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +3. Select **Service Actions > Activate HAWQ Standby Master** to start the Activate HAWQ Standby Master Wizard. +4. Read the description of the Wizard and click **Next** to review the tasks that will be performed. +5. Ambari displays the host name of the current HAWQ Master that will be removed from the cluster, as well as the HAWQ Standby Master host that will be activated. The information is provided only for review and cannot be edited on this page. Click **Next** to confirm the operation. +6. Click **OK** to confirm that you want to perform the procedure, as it is not possible to roll back the operation using Ambari. + + Ambari displays a list of tasks that are performed to activate the standby server and remove the previous HAWQ Master host. Click on any of the tasks to view progress or to view the actual log messages that are generated while performing the task. +7. Click **Complete** after the Wizard finishes all tasks. + + **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby Master](#amb-add-standby) to configure a new one. + +## <a id="amb-add-standby"></a>Adding a HAWQ Standby Master + +The HAWQ Standby Master serves as a backup of the HAWQ Master host, and is an important part of providing high availability for the HAWQ cluster. When your cluster uses a standby master, you can activate the standby if the active HAWQ Master host fails or becomes unreachable. + +### When to Perform +* Execute this procedure during a scheduled maintenance period, because it requires a restart of the HAWQ service. +* Adding a HAWQ standby master is recommended as a best practice for all new clusters to provide high availability. +* Add a new standby master soon after you activate an existing standby master to ensure that the cluster has a backup master service. + +### Procedure + +1. Select an existing host in the cluster to run the HAWQ standby master. You cannot run the standby master on the same host that runs the HAWQ master. Also, do not run a standby master on the node where you deployed the Ambari server; if the Ambari postgres instance is running on the same port as the HAWQ master posgres instance, initialization fails and will leave the cluster in an inconsistent state. +1. Login to the HAWQ host that you chose to run the standby master and determine if there is an existing HAWQ master directory (for example, `/data/hawq/master`) on the machine. If the directory exists, rename the directory. For example: + + ```shell + $ mv /data/hawq/master /data/hawq/master-old + ``` + + **Note:** If a HAWQ master directory exists on the host when you configure the HAWQ standby master, then the standby master may be initialized with stale data. Rename any existing master directory before you proceed. + +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +3. Select **Service Actions > Add HAWQ Standby Master** to start the Add HAWQ Standby Master Wizard. +4. Read the Get Started page for information about HAWQ the standby master and to acknowledge that the procedure requires a service restart. Click **Next** to display the Select Host page. +5. Use the dropdown menu to select a host to use for the HAWQ Standby Master. Click **Next** to display the Review page. + + **Note:** + * The Current HAWQ Master host is shown only for reference. You cannot change the HAWQ Master host when you configure a standby master. + * You cannot place the standby master on the same host as the HAWQ master. +6. Review the information to verify the host on which the HAWQ Standby Master will be installed. Click **Back** to change your selection or **Next** to continue. +7. Confirm that you have renamed any existing HAWQ master data directory on the selected host machine, as described earlier in this procedure. If an existing master data directory exists, the new HAWQ Standby Master may be initialized with stale data and can place the cluster in an inconsistent state. Click **Confirm** to continue. + + Ambari displays a list of tasks that are performed to install the standby master server and reconfigure the cluster. Click on any of the tasks to view progress or to view the actual log messages that are generated while performing the task. +7. Click **Complete** after the Wizard finishes all tasks. + +## <a id="amb-remove-standby"></a>Removing the HAWQ Standby Master + +This service action enables you to remove the HAWQ Standby Master component in situations where you may need to reinstall the component. + +### When to Perform +* Execute this procedure if you need to decommission or replace theHAWQ Standby Master host. +* Execute this procedure and then add the HAWQ Standby Master once again, if the HAWQ Standby Master is unable to synchronize with the HAWQ Master and you need to reinitialize the service. +* Execute this procedure during a scheduled maintenance period, because it requires a restart of the HAWQ service. + +### Procedure +1. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) +2. Click **HAWQ** in the list of installed services. +3. Select **Service Actions > Remove HAWQ Standby Master** to start the Remove HAWQ Standby Master Wizard. +4. Read the Get Started page for information about the procedure and to acknowledge that the procedure requires a service restart. Click **Next** to display the Review page. +5. Ambari displays the HAWQ Standby Master host that will be removed from the cluster configuration. Click **Next** to continue, then click **OK** to confirm. + + Ambari displays a list of tasks that are performed to remove the standby master from the cluster. Click on any of the tasks to view progress or to view the actual log messages that are generated while performing the task. + +7. Click **Complete** after the Wizard finishes all tasks. + + **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby Master](#amb-add-standby) to configure a new one. + +## <a id="hdp-upgrade"></a>Upgrading the HDP Stack + +If you install HAWQ using Ambari 2.2.2 with the HDP 2.3 stack, before you attempt to upgrade to HDP 2.4 you must use Ambari to change the `dfs.allow.truncate` property to `false`. Ambari will display a configuration warning with this setting, but it is required in order to complete the upgrade; choose **Proceed Anyway** when Ambari warns you about the configured value of `dfs.allow.truncate`. + +After you complete the upgrade to HDP 2.4, change the value of `dfs.allow.truncate` back to `true` to ensure that HAWQ can operate as intended. + +## <a id="gpadmin-password-change"></a>Changing the HAWQ gpadmin Password +The password issued by the Ambari web console is used for the `hawq ssh-exkeys` utility, which is run during the start phase of the HAWQ Master. +Ambari stores and uses its own copy of the gpadmin password, independently of the host system. Passwords on the master and slave nodes are not automatically updated and synchronized with Ambari. Not updating the Ambari system user password causes Ambari to behave as if the gpadmin password was never changed \(it keeps using the old password\). + +If passwordless ssh has not been set up, `hawq ssh-exkeys` attempts to exchange the key by using the password provided by the Ambari web console. If the password on the host machine differs from the HAWQ System User password recognized on Ambari, exchanging the key with the HAWQ Master fails. Components without passwordless ssh might not be registered with the HAWQ cluster. + +### When to Perform +You should change the gpadmin password when: + +* The gpadmin password on the host machines has expired. +* You want to change passwords as part of normal system security procedures. +When updating the gpadmin password, it must be kept in synch with the gpadmin user on the HAWQ hosts. This requires manually changing the password on the Master and Slave hosts, then updating the Ambari password. + +###Procedure +All of the listed steps are mandatory. This ensures that HAWQ service remains fully functional. + +1. Use a script to manually change the password for the gpadmin user on all HAWQ hosts \(all Master and Slave component hosts\). To manually update the password, you must have ssh access to all host machines as the gpadmin user. Generate a hosts file to use with the `hawq ssh` command to reset the password on all hosts. Use a text editor to create a file that lists the hostname of the master node, the standby master node, and each segment node used in the cluster. Specify one hostname per line, for example: + + ``` + mdw + smdw + sdw1 + sdw2 + sdw3 + ``` + + You can then use a command similar to the following to change the password on all hosts that are listed in the file: + + ```shell + $ hawq ssh -f hawq_hosts 'echo "gpadmin:newpassword" | /usr/sbin/chpasswd' + ``` + + **Note:** Be sure to make appropriate user and password system administrative changes in order to prevent operational disruption. For example, you may need to disable the password expiration policy for the `gpadmin` account. +2. Access the Ambari web console at http://ambari.server.hostname:8080, and login as the "admin" user. \(The default password is also "admin".\) Then perform the following steps: + 1. Click **HAWQ** in the list of installed services. + 2. On the HAWQ Server Configs page, go to the **Advanced** tab and update the **HAWQ System User Password** to the new password specified in the script. + 3. Click **Save** to save the updated configuration. + 4. Restart HAWQ service to propagate the configuration change to all Ambari agents. + + This will synchronize the password on the host machines with the password that you specified in Ambari. + +## <a id="gpadmin-setup-alert"></a>Setting Up Alerts + +Alerts advise you of when a HAWQ process is down or not responding, or when certain conditions requiring attention occur. +Alerts can be created for the Master, Standby Master, Segments, and PXF components. You can also set up custom alert groups to monitor these conditions and send email notifications when they occur. + +### When to Perform +Alerts are enabled by default. You might want to disable alert functions when performing system operations in maintenance mode and then re-enable them after returning to normal operation. + +You can configure alerts to display messages for all system status changes or only for conditions of interest, such as warnings or critical conditions. Alerts can advise you if there are communication issues between the HAWQ Master and HAWQ segments, or if the HAWQ Master, Standby Master, a segment, or the PXF service is down or not responding. + +You can configure Ambari to check for alerts at specified intervals, on a particular service or host, and what level of criticality you want to trigger an alert (OK, WARNING, or CRITICAL). + +### Procedure +Ambari can show Alerts and also configure certain status conditions. + +#### Viewing Alerts +To view the current alert information for HAWQ, click the **Groups** button at the top left of the Alerts page, then select **HAWQ Default** in the drop-down menu, then click on the **Alert** button at the top of the Ambari console. Ambari will display a list of all available alert functions and their current status. + +To check PXF alerts, click the **Groups** dropdown button at the top left of the Alerts page. Select **PXF Default** in the dropdown menu. Alerts are displayed on the PXF Status page. + +To view the current Alert settings, click on the name of the alert. + +The Alerts you can view are as follows: + +* HAWQ Master Process: +This alert is triggered when the HAWQ Master process is down or not responding. + +* HAWQ Segment Process: +This alert is triggered when a HAWQ Segment on a node is down or not responding. + +* HAWQ Standby Master Process: +This alert is triggered when the HAWQ Standby Master process is down or not responding. If no standby is present, the Alert shows as **NONE**. + +* HAWQ Standby Master Sync Status: +This alert is triggered when the HAWQ Standby Master is not synchronized with the HAWQ Master. Using this Alert eliminates the need to check the gp\_master\_mirroring catalog table to determine if the Standby Master is fully synchronized. +If no standby Master is present, the status will show as **UNKNOWN**. + If this Alert is triggered, go to the HAWQ **Services** tab and click on the **Service Action** button to re-sync the HAWQ Standby Master with the HAWQ Master. + +* HAWQ Segment Registration Status: +This alert is triggered when any of the HAWQ Segments fail to register with the HAWQ Master. This indicates that the HAWQ segments having an up status in the gp\_segment\_configuration table do not match the HAWQ Segments listed in the /usr/local/hawq/etc/slaves file on the HAWQ Master. + +* Percent HAWQ Segment Status Available: +This Alert monitors the percentage of HAWQ segments available versus total segments. + Alerts for **WARN**, and **CRITICAL** are displayed when the number of unresponsive HAWQ segments in the cluster is greater than the specified threshold. Otherwise, the status will show as **OK**. + +* PXF Process Alerts: +PXF Process alerts are triggered when a PXF process on a node is down or not responding on the network. If PXF Alerts are enabled, the Alert status is shown on the PXF Status page. + +#### Setting the Monitoring Inteval +You can customize how often you wish the system to check for certain conditions. The default interval for checking the HAWQ system is 1 minute. + +To customize the interval, perform the following steps: + +1. Click on the name of the Alert you want to edit. +2. When the Configuration screen appears, click **Edit**. +3. Enter a number for how often to check status for the selected Alert, then click **Save**. The interval must be specified in whole minutes. + + +#### Setting the Available HAWQ Segment Threshold +HAWQ monitors the percentage of available HAWQ segments and can send an alert when a specified percent of unresponsive segments is reached. + +To set the threshold for the unresponsive segments that will trigger an alert: + + 1. Click on **Percent HAWQ Segments Available**. + 2. Click **Edit**. Enter the percentage of total segments to create a **Warning** alert (default is 10 percent of the total segments) or **Critical** alert (default is 25 percent of total segments). + 3. Click **Save** when done. + Alerts for **WARN**, and **CRITICAL** will be displayed when the number of unresponsive HAWQ segments in the cluster is greater than the specified percentage. +
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/admin/ambari-rest-api.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/admin/ambari-rest-api.html.md.erb b/markdown/admin/ambari-rest-api.html.md.erb new file mode 100644 index 0000000..2cc79e4 --- /dev/null +++ b/markdown/admin/ambari-rest-api.html.md.erb @@ -0,0 +1,163 @@ +--- +title: Using the Ambari REST API +--- + +You can monitor and manage the resources in your HAWQ cluster using the Ambari REST API. In addition to providing access to the metrics information in your cluster, the API supports viewing, creating, deleting, and updating cluster resources. + +This section will provide an introduction to using the Ambari REST APIs for HAWQ-related cluster management activities. + +Refer to [Ambari API Reference v1](https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md) for the official Ambari API documentation, including full REST resource definitions and response semantics. *Note*: These APIs may change in new versions of Ambari. + + +## <a id="ambari-rest-uri"></a>Manageable HAWQ Resources + +HAWQ provides several REST resources to support starting and stopping services, executing service checks, and viewing configuration information among other activities. HAWQ resources you can manage using the Ambari REST API include: + +| Ambari Resource | Description | +|----------------------|------------------------| +| cluster | The HAWQ cluster. | +| service | The HAWQ and PXF service. You can manage other Hadoop services as well. | +| component | A specific HAWQ/PXF service component, i.e. the HAWQ Master, PXF. | +| configuration | A specific HAWQ/PXF configuration entity, for example the hawq-site or pxf-profiles configuration files, or a specific single HAWQ or PXF configuration property. | +| request | A group of tasks. | + +## <a id="ambari-rest-uri"></a>URI Structure + +The Ambari REST API provides access to HAWQ cluster resources via URI (uniform resource identifier) paths. To use the Ambari REST API, you will send HTTP requests and parse JSON-formatted HTTP responses. + +The Ambari REST API supports standard HTTP request methods including: + +- `GET` - read resource properties, metrics +- `POST` - create new resource +- `PUT` - update resource +- `DELETE` - delete resource + +URIs for Ambari REST API resources have the following structure: + +``` shell +http://<ambari-server-host>:<port>/api/v1/<resource-path> +``` + +The Ambari REST API supports the following HAWQ-related \<resource-paths\>: + +| REST Resource Path | Description | +|----------------------|------------------------| +| clusters/\<cluster\-name\> | The HAWQ cluster name. | +| clusters/\<cluster\-name\>/services/PXF | The PXF service. | +| clusters/\<cluster\-name\>/services/HAWQ | The HAWQ service. | +| clusters/\<cluster\-name\>/services/HAWQ/components | All HAWQ service components. | +| clusters/\<cluster\-name\>/services/HAWQ/components/\<name\> | A specific HAWQ service component, i.e. HAWQMASTER. | +| clusters/\<cluster\-name\>/configurations | Cluster configurations. | +| clusters/\<cluster\-name\>/requests | Group of tasks that run a command. | + +## <a id="ambari-rest-curl"></a>Submitting Requests with cURL + +Your HTTP request to the Ambari REST API should include the following information: + +- User name and password for basic authentication. +- An HTTP request header. +- The HTTP request method. +- JSON-formatted request data, if required. +- The URI identifying the Ambari REST resource. + +You can use the `curl` command to transfer HTTP request data to, and receive data from, the Ambari server using the HTTP protocol. + +Use the following syntax to issue a `curl` command for Ambari HAWQ/PXF management operations: + +``` shell +$ curl -u <user>:<passwd> -H <header> -X GET|POST|PUT|DELETE -d <data> <URI> +``` + +`curl` options relevant to Ambari REST API communication include: + +| Option | Description | +|----------------------|------------------------| +| -u \<user\>:\<passwd\> | Identify the username and password for basic authentication to the HTTP server. | +| -H \<header\> | Identify an extra header to include in the HTTP request. \<header\> must specify `'X-Requested-By:ambari'`. | +| -X \<command\> | Identify the request method. \<command\> may specify `GET` (the default), `POST`, `PUT`, and `DELETE`. | +| -d \<data\> | Send the specified \<data\> to the HTTP server along with the request. The \<command\> and \<URI\> determine if \<data\> is required, and if so, its content. | +| \<URI\> | Path to the Ambari REST resource. | + + +## <a id="ambari-rest-api-auth"></a>Authenticating with the Ambari REST API + +The first step in using the Ambari REST API is to authenticate with the Ambari server. The Ambari REST API supports HTTP basic authentication. With this authentication method, you provide a username and password that is internally encoded and sent in the HTTP header. + +Example: Testing Authentication + +1. Set up some environment variables; replace the values with those appropriate for your operating environment. For example: + + ``` shell + $ export AMBUSER=admin + $ export AMBPASSWD=admin + $ export AMBHOST=<ambari-server> + $ export AMBPORT=8080 + ``` + +2. Submit a `curl` request to the Ambari server: + + ``` shell + $ curl -u $AMBUSER:$AMBPASSWD http://$AMBHOST:$AMBPORT + ``` + + If authentication succeeds, Apache license information is displayed. + + +## <a id="ambari-rest-using"></a>Using the Ambari REST API for HAWQ Management + + +### <a id="ambari-rest-ex-clustname"></a>Example: Retrieving the HAWQ Cluster Name + +1. Set up an additional environment variables: + + ``` shell + $ export AMBCREDS="$AMBUSER:$AMBPASSWD" + $ export AMBURLBASE="http://${AMBHOST}:${AMBPORT}/api/v1/clusters"; + ``` + + You will use these variables in upcoming examples to simplify `curl` calls. + +2. Use the Ambari REST API to determine the name of your HAWQ cluster; also set `$AMBURLBASE` to include the cluster name: + + ``` shell + $ export CLUSTER_NAME="$(curl -u ${AMBCREDS} -i -H 'X-Requested-By:ambari' $AMBURLBASE | sed -n 's/.*"cluster_name" : "\([^\"]*\)".*/\1/p')" + $ echo $CLUSTER_NAME + TestCluster + $ export AMBURLBASE=$AMBURLBASE/$CLUSTER_NAME + ``` + +### <a id="ambari-rest-ex-mgmt"></a>Examples: Managing the HAWQ and PXF Services + +The following subsections provide `curl` commands for common HAWQ cluster management activities. + +Refer to [API usage scenarios, troubleshooting, and other FAQs](https://cwiki.apache.org/confluence/display/AMBARI/API+usage+scenarios%2C+troubleshooting%2C+and+other+FAQs) for additional Ambari REST API usage examples. + + +#### <a id="ambari-rest-ex-get"></a>Viewing HAWQ Cluster Service and Configuration Information + +| Task |Command | +|----------------------|------------------------| +| View HAWQ service information. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' $AMBURLBASE/services/HAWQ` | +| List all HAWQ components. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' $AMBURLBASE/services/HAWQ/components` | +| View information about the HAWQ master. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' $AMBURLBASE/services/HAWQ/components/HAWQMASTER` | +| View the `hawq-site` configuration settings. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' "$AMBURLBASE/configurations?type=hawq-site&tag=TOPOLOGY_RESOLVED"` | +| View the initial `core-site` configuration settings. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' "$AMBURLBASE/configurations?type=core-site&tag=INITIAL"` | +| View the `pxf-profiles` configuration file. | `curl -u $AMBCREDS -X GET -H 'X-Requested-By:ambari' "$AMBURLBASE/configurations?type=pxf-profiles&tag=INITIAL"` | +| View all components on node. | `curl -u $AMBCREDS -i -X GET -H 'X-Requested-B:ambari' $AMBURLBASE/hosts/<hawq-node>` | + + +#### <a id="ambari-rest-ex-put"></a>Starting/Stopping HAWQ and PXF Services + +| Task |Command | +|----------------------|------------------------| +| Start the HAWQ service. | `curl -u $AMBCREDS -X PUT -H 'X-Requested-By:ambari' -d '{"RequestInfo": {"context" :"Start HAWQ via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' $AMBURLBASE/services/HAWQ` | +| Stop the HAWQ service. | `curl -u $AMBCREDS -X PUT -H 'X-Requested-By:ambari' -d '{"RequestInfo": {"context" :"Stop HAWQ via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' $AMBURLBASE/services/HAWQ` | +| Start the PXF service. | `curl -u $AMBCREDS -X PUT -H 'X-Requested-By:ambari' -d '{"RequestInfo": {"context" :"Start PXF via REST"}, "Body": {"ServiceInfo": {"state": "STARTED"}}}' $AMBURLBASE//services/PXF` | +| Stop the PXF service. | `curl -u $AMBCREDS -X PUT -H 'X-Requested-By:ambari' -d '{"RequestInfo": {"context" :"Stop PXF via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' $AMBURLBASE/services/PXF` | + +#### <a id="ambari-rest-ex-post"></a>Invoking HAWQ and PXF Service Actions + +| Task |Command | +|----------------------|------------------------| +| Run a HAWQ service check. | `curl -u $AMBCREDS -X POST -H 'X-Requested-By:ambari' -d '{"RequestInfo":{"context":"HAWQ Service Check","command":"HAWQ_SERVICE_CHECK"}, "Requests/resource_filters":[{ "service_name":"HAWQ"}]}' $AMBURLBASE/requests` | +| Run a PXF service check. | `curl -u $AMBCREDS -X POST -H 'X-Requested-By:ambari' -d '{"RequestInfo":{"context":"PXF Service Check","command":"PXF_SERVICE_CHECK"}, "Requests/resource_filters":[{ "service_name":"PXF"}]}' $AMBURLBASE/requests` | http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/admin/maintain.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/admin/maintain.html.md.erb b/markdown/admin/maintain.html.md.erb new file mode 100644 index 0000000..f4b1491 --- /dev/null +++ b/markdown/admin/maintain.html.md.erb @@ -0,0 +1,31 @@ +--- +title: Routine System Maintenance Tasks +--- + +## <a id="overview-topic"></a>Overview + +To keep a HAWQ system running efficiently, the database must be regularly cleared of expired data and the table statistics must be updated so that the query optimizer has accurate information. + +HAWQ requires that certain tasks be performed regularly to achieve optimal performance. The tasks discussed here are required, but database administrators can automate them using standard UNIX tools such as `cron` scripts. An administrator sets up the appropriate scripts and checks that they execute successfully. See [Recommended Monitoring and Maintenance Tasks](RecommendedMonitoringTasks.html) for additional suggested maintenance activities you can implement to keep your HAWQ system running optimally. + +## <a id="topic10"></a>Database Server Log Files + +HAWQ log output tends to be voluminous, especially at higher debug levels, and you do not need to save it indefinitely. Administrators rotate the log files periodically so new log files are started and old ones are removed. + +HAWQ has log file rotation enabled on the master and all segment instances. Daily log files are created in the `pg_log` subdirectory of the master and each segment data directory using the following naming convention: <code>hawq-<i>YYYY-MM-DD\_hhmmss</i>.csv</code>. Although log files are rolled over daily, they are not automatically truncated or deleted. Administrators need to implement scripts or programs to periodically clean up old log files in the `pg_log` directory of the master and of every segment instance. + +For information about viewing the database server log files, see [Viewing the Database Server Log Files](monitor.html). + +## <a id="topic11"></a>Management Utility Log Files + +Log files for the HAWQ management utilities are written to `~/hawqAdminLogs` by default. The naming convention for management log files is: + +<pre><code><i>script_name_date</i>.log +</code></pre> + +The log entry format is: + +<pre><code><i>timestamp:utility:host:user</i>:[INFO|WARN|FATAL]:<i>message</i> +</code></pre> + +The log file for a particular utility execution is appended to its daily log file each time that utility is run. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/admin/monitor.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/admin/monitor.html.md.erb b/markdown/admin/monitor.html.md.erb new file mode 100644 index 0000000..418c8c3 --- /dev/null +++ b/markdown/admin/monitor.html.md.erb @@ -0,0 +1,444 @@ +--- +title: Monitoring a HAWQ System +--- + +You can monitor a HAWQ system using a variety of tools included with the system or available as add-ons. + +Observing the HAWQ system day-to-day performance helps administrators understand the system behavior, plan workflow, and troubleshoot problems. This chapter discusses tools for monitoring database performance and activity. + +Also, be sure to review [Recommended Monitoring and Maintenance Tasks](RecommendedMonitoringTasks.html) for monitoring activities you can script to quickly detect problems in the system. + + +## <a id="topic31"></a>Using hawq\_toolkit + +Use HAWQ's administrative schema [*hawq\_toolkit*](../reference/toolkit/hawq_toolkit.html) to query the system catalogs, log files, and operating environment for system status information. The *hawq\_toolkit* schema contains several views you can access using SQL commands. The *hawq\_toolkit* schema is accessible to all database users. Some objects require superuser permissions. Use a command similar to the following to add the *hawq\_toolkit* schema to your schema search path: + +```sql +=> SET ROLE 'gpadmin' ; +=# SET search_path TO myschema, hawq_toolkit ; +``` + +## <a id="topic3"></a>Monitoring System State + +As a HAWQ administrator, you must monitor the system for problem events such as a segment going down or running out of disk space on a segment host. The following topics describe how to monitor the health of a HAWQ system and examine certain state information for a HAWQ system. + +- [Checking System State](#topic12) +- [Checking Disk Space Usage](#topic15) +- [Viewing Metadata Information about Database Objects](#topic24) +- [Viewing Query Workfile Usage Information](#topic27) + +### <a id="topic12"></a>Checking System State + +A HAWQ system is comprised of multiple PostgreSQL instances \(the master and segments\) spanning multiple machines. To monitor a HAWQ system, you need to know information about the system as a whole, as well as status information of the individual instances. The `hawq state` utility provides status information about a HAWQ system. + +#### <a id="topic13"></a>Viewing Master and Segment Status and Configuration + +The default `hawq state` action is to check segment instances and show a brief status of the valid and failed segments. For example, to see a quick status of your HAWQ system: + +```shell +$ hawq state -b +``` + +You can also display information about the HAWQ master data directory by invoking `hawq state` with the `-d` option: + +```shell +$ hawq state -d <master_data_dir> +``` + + +### <a id="topic15"></a>Checking Disk Space Usage + +#### <a id="topic16"></a>Checking Sizing of Distributed Databases and Tables + +The *hawq\_toolkit* administrative schema contains several views that you can use to determine the disk space usage for a distributed HAWQ database, schema, table, or index. + +##### <a id="topic17"></a>Viewing Disk Space Usage for a Database + +To see the total size of a database \(in bytes\), use the *hawq\_size\_of\_database* view in the *hawq\_toolkit* administrative schema. For example: + +```sql +=> SELECT * FROM hawq_toolkit.hawq_size_of_database + ORDER BY sodddatname; +``` + +##### <a id="topic18"></a>Viewing Disk Space Usage for a Table + +The *hawq\_toolkit* administrative schema contains several views for checking the size of a table. The table sizing views list the table by object ID \(not by name\). To check the size of a table by name, you must look up the relation name \(`relname`\) in the *pg\_class* table. For example: + +```sql +=> SELECT relname AS name, sotdsize AS size, sotdtoastsize + AS toast, sotdadditionalsize AS other + FROM hawq_toolkit.hawq_size_of_table_disk AS sotd, pg_class + WHERE sotd.sotdoid=pg_class.oid ORDER BY relname; +``` + +##### <a id="topic19"></a>Viewing Disk Space Usage for Indexes + +The *hawq\_toolkit* administrative schema contains a number of views for checking index sizes. To see the total size of all index\(es\) on a table, use the *hawq\_size\_of\_all\_table\_indexes* view. To see the size of a particular index, use the *hawq\_size\_of\_index* view. The index sizing views list tables and indexes by object ID \(not by name\). To check the size of an index by name, you must look up the relation name \(`relname`\) in the *pg\_class* table. For example: + +```sql +=> SELECT soisize, relname AS indexname + FROM pg_class, hawq_size_of_index + WHERE pg_class.oid=hawq_size_of_index.soioid + AND pg_class.relkind='i'; +``` + +### <a id="topic24"></a>Viewing Metadata Information about Database Objects + +HAWQ uses its system catalogs to track various metadata information about the objects stored in a database (tables, views, indexes and so on), as well as global objects including roles and tablespaces. + +#### <a id="topic25"></a>Viewing the Last Operation Performed + +You can use the system views *pg\_stat\_operations* and *pg\_stat\_partition\_operations* to look up actions performed on a database object. For example, to view when the `cust` table was created and when it was last analyzed: + +```sql +=> SELECT schemaname AS schema, objname AS table, + usename AS role, actionname AS action, + subtype AS type, statime AS time + FROM pg_stat_operations + WHERE objname='cust'; +``` + +``` +Â schema | table | role | action | type | time +--------+-------+------+---------+-------+-------------------------- +Â Â sales | cust | main | CREATE | TABLE | 2010-02-09 18:10:07.867977-08 +Â Â sales | cust | main | VACUUM | | 2010-02-10 13:32:39.068219-08 +Â Â sales | cust | main | ANALYZE | | 2010-02-25 16:07:01.157168-08 +(3 rows) + +``` + +#### <a id="topic26"></a>Viewing the Definition of an Object + +You can use the `psql` `\d` meta-command to display the definition of an object, such as a table or view. For example, to see the definition of a table named `sales`: + +``` sql +=> \d sales +``` + +``` +Append-Only Table "public.sales" + Column | Type | Modifiers +--------+---------+----------- + id | integer | + year | integer | + qtr | integer | + day | integer | + region | text | +Compression Type: None +Compression Level: 0 +Block Size: 32768 +Checksum: f +Distributed by: (id) +``` + + +### <a id="topic27"></a>Viewing Query Workfile Usage Information + +The HAWQ administrative schema *hawq\_toolkit* contains views that display information about HAWQ workfiles. HAWQ creates workfiles on disk if it does not have sufficient memory to execute the query in memory. This information can be used for troubleshooting and tuning queries. The information in the views can also be used to specify the values for the HAWQ configuration parameters `hawq_workfile_limit_per_query` and `hawq_workfile_limit_per_segment`. + +Views in the *hawq\_toolkit* schema include: + +- *hawq\_workfile\_entries* - one row for each operator currently using disk space for workfiles on a segment +- *hawq\_workfile\_usage\_per\_query* - one row for each running query currently using disk space for workfiles on a segment +- *hawq\_workfile\_usage\_per\_segment* - one row for each segment where each row displays the total amount of disk space currently in use for workfiles on the segment + + +## <a id="topic28"></a>Viewing the Database Server Log Files + +Every database instance in HAWQ \(master and segments\) runs a PostgreSQL database server with its own server log file. Daily log files are created in the `pg_log` directory of the master and each segment data directory. + +### <a id="topic29"></a>Log File Format + +The server log files are written in comma-separated values \(CSV\) format. Log entries may not include values for all log fields. For example, only log entries associated with a query worker process will have the `slice_id` populated. You can identify related log entries of a particular query by the query's session identifier \(`gp_session_id`\) and command identifier \(`gp_command_count`\). + +Log entries may include the following fields: + +<table> + <tr><th>#</th><th>Field Name</th><th>Data Type</th><th>Description</th></tr> + <tr><td>1</td><td>event_time</td><td>timestamp with time zone</td><td>Time that the log entry was written to the log</td></tr> + <tr><td>2</td><td>user_name</td><td>varchar(100)</td><td>The database user name</td></tr> + <tr><td>3</td><td>database_name</td><td>varchar(100)</td><td>The database name</td></tr> + <tr><td>4</td><td>process_id</td><td>varchar(10)</td><td>The system process ID (prefixed with "p")</td></tr> + <tr><td>5</td><td>thread_id</td><td>varchar(50)</td><td>The thread count (prefixed with "th")</td></tr> + <tr><td>6</td><td>remote_host</td><td>varchar(100)</td><td>On the master, the hostname/address of the client machine. On the segment, the hostname/address of the master.</td></tr> + <tr><td>7</td><td>remote_port</td><td>varchar(10)</td><td>The segment or master port number</td></tr> + <tr><td>8</td><td>session_start_time</td><td>timestamp with time zone</td><td>Time session connection was opened</td></tr> + <tr><td>9</td><td>transaction_id</td><td>int</td><td>Top-level transaction ID on the master. This ID is the parent of any subtransactions.</td></tr> + <tr><td>10</td><td>gp_session_id</td><td>text</td><td>Session identifier number (prefixed with "con")</td></tr> + <tr><td>11</td><td>gp_command_count</td><td>text</td><td>The command number within a session (prefixed with "cmd")</td></tr> + <tr><td>12</td><td>gp_segment</td><td>text</td><td>The segment content identifier. The master always has a content ID of -1.</td></tr> + <tr><td>13</td><td>slice_id</td><td>text</td><td>The slice ID (portion of the query plan being executed)</td></tr> + <tr><td>14</td><td>distr_tranx_id</td><td>text</td><td>Distributed transaction ID</td></tr> + <tr><td>15</td><td>local_tranx_id</td><td>text</td><td>Local transaction ID</td></tr> + <tr><td>16</td><td>sub_tranx_id</td><td>text</td><td>Subtransaction ID</td></tr> + <tr><td>17</td><td>event_severity</td><td>varchar(10)</td><td>Values include: LOG, ERROR, FATAL, PANIC, DEBUG1, DEBUG2</td></tr> + <tr><td>18</td><td>sql_state_code</td><td>varchar(10)</td><td>SQL state code associated with the log message</td></tr> + <tr><td>19</td><td>event_message</td><td>text</td><td>Log or error message text</td></tr> + <tr><td>20</td><td>event_detail</td><td>text</td><td>Detail message text associated with an error or warning message</td></tr> + <tr><td>21</td><td>event_hint</td><td>text</td><td>Hint message text associated with an error or warning message</td></tr> + <tr><td>22</td><td>internal_query</td><td>text</td><td>The internally-generated query text</td></tr> + <tr><td>23</td><td>internal_query_pos</td><td>int</td><td>The cursor index into the internally-generated query text</td></tr> + <tr><td>24</td><td>event_context</td><td>text</td><td>The context in which this message gets generated</td></tr> + <tr><td>25</td><td>debug_query_string</td><td>text</td><td>User-supplied query string with full detail for debugging. This string can be modified for internal use.</td></tr> + <tr><td>26</td><td>error_cursor_pos</td><td>int</td><td>The cursor index into the query string</td></tr> + <tr><td>27</td><td>func_name</td><td>text</td><td>The function in which this message is generated</td></tr> + <tr><td>28</td><td>file_name</td><td>text</td><td>The internal code file where the message originated</td></tr> + <tr><td>29</td><td>file_line</td><td>int</td><td>The line of the code file where the message originated</td></tr> + <tr><td>30</td><td>stack_trace</td><td>text</td><td>Stack trace text associated with this message</td></tr> +</table> +### <a id="topic30"></a>Searching the HAWQ Server Log Files + +You can use the `gplogfilter` HAWQ utility to search through a HAWQ log file for entries matching specific criteria. By default, this utility searches through the HAWQ master log file in the default logging location. For example, to display the entries to the master log file starting after 2 pm on a certain date: + +``` shell +$ gplogfilter -b '2016-01-18 14:00' +``` + +To search through all segment log files simultaneously, run `gplogfilter` through the `hawq ssh` utility. For example, specify a \<seg\_hosts\> file that includes all segment hosts of interest, then invoke `gplogfilter` to display the last three lines of each segment log file on each segment host. (Note: enter the commands after the `=>` prompt, do not include the `=>`.): + +``` shell +$ hawq ssh -f <seg_hosts> +=> source /usr/local/hawq/greenplum_path.sh +=> gplogfilter -n 3 /data/hawq/segment/pg_log/hawq*.csv +``` + +## <a id="topic_jx2_rqg_kp"></a>HAWQ Error Codes + +The following section describes SQL error codes for certain database events. + +### <a id="topic_pyh_sqg_kp"></a>SQL Standard Error Codes + +The following table lists all the defined error codes. Some are not used, but are defined by the SQL standard. The error classes are also shown. For each error class there is a standard error code having the last three characters 000. This code is used only for error conditions that fall within the class but do not have any more-specific code assigned. + +The PL/pgSQL condition name for each error code is the same as the phrase shown in the table, with underscores substituted for spaces. For example, code 22012, DIVISION BY ZERO, has condition name DIVISION\_BY\_ZERO. Condition names can be written in either upper or lower case. + +**Note:** PL/pgSQL does not recognize warning, as opposed to error, condition names; those are classes 00, 01, and 02. + +|Error Code|Meaning|Constant| +|----------|-------|--------| +|**Class 00**â Successful Completion| +|00000|SUCCESSFUL COMPLETION|successful\_completion| +|Class 01 â Warning| +|01000|WARNING|warning| +|0100C|DYNAMIC RESULT SETS RETURNED|dynamic\_result\_sets\_returned| +|01008|IMPLICIT ZERO BIT PADDING|implicit\_zero\_bit\_padding| +|01003|NULL VALUE ELIMINATED IN SET FUNCTION|null\_value\_eliminated\_in\_set\_function| +|01007|PRIVILEGE NOT GRANTED|privilege\_not\_granted| +|01006|PRIVILEGE NOT REVOKED|privilege\_not\_revoked| +|01004|STRING DATA RIGHT TRUNCATION|string\_data\_right\_truncation| +|01P01|DEPRECATED FEATURE|deprecated\_feature| +|**Class 02** â No Data \(this is also a warning class per the SQL standard\)| +|02000|NO DATA|no\_data| +|02001|NO ADDITIONAL DYNAMIC RESULT SETS RETURNED|no\_additional\_dynamic\_result\_sets\_returned| +|**Class 03** â SQL Statement Not Yet Complete| +|03000|SQL STATEMENT NOT YET COMPLETE|sql\_statement\_not\_yet\_complete| +|**Class 08** â Connection Exception| +|08000|CONNECTION EXCEPTION|connection\_exception| +|08003|CONNECTION DOES NOT EXIST|connection\_does\_not\_exist| +|08006|CONNECTION FAILURE|connection\_failure| +|08001|SQLCLIENT UNABLE TO ESTABLISH SQLCONNECTION|sqlclient\_unable\_to\_establish\_sqlconnection| +|08004|SQLSERVER REJECTED ESTABLISHMENT OF SQLCONNECTION|sqlserver\_rejected\_establishment\_of\_sqlconnection| +|08007|TRANSACTION RESOLUTION UNKNOWN|transaction\_resolution\_unknown| +|08P01|PROTOCOL VIOLATION|protocol\_violation| +|**Class 09** â Triggered Action Exception| +|09000|TRIGGERED ACTION EXCEPTION|triggered\_action\_exception| +|**Class 0A** â Feature Not Supported| +|0A000|FEATURE NOT SUPPORTED|feature\_not\_supported| +|**Class 0B** â Invalid Transaction Initiation| +|0B000|INVALID TRANSACTION INITIATION|invalid\_transaction\_initiation| +|**Class 0F** â Locator Exception| +|0F000|LOCATOR EXCEPTION|locator\_exception| +|0F001|INVALID LOCATOR SPECIFICATION|invalid\_locator\_specification| +|**Class 0L** â Invalid Grantor| +|0L000|INVALID GRANTOR|invalid\_grantor| +|0LP01|INVALID GRANT OPERATION|invalid\_grant\_operation| +|**Class 0P** â Invalid Role Specification| +|0P000|INVALID ROLE SPECIFICATION|invalid\_role\_specification| +|**Class 21** â Cardinality Violation| +|21000|CARDINALITY VIOLATION|cardinality\_violation| +|**Class 22** â Data Exception| +|22000|DATA EXCEPTION|data\_exception| +|2202E|ARRAY SUBSCRIPT ERROR|array\_subscript\_error| +|22021|CHARACTER NOT IN REPERTOIRE|character\_not\_in\_repertoire| +|22008|DATETIME FIELD OVERFLOW|datetime\_field\_overflow| +|22012|DIVISION BY ZERO|division\_by\_zero| +|22005|ERROR IN ASSIGNMENT|error\_in\_assignment| +|2200B|ESCAPE CHARACTER CONFLICT|escape\_character\_conflict| +|22022|INDICATOR OVERFLOW|indicator\_overflow| +|22015|INTERVAL FIELD OVERFLOW|interval\_field\_overflow| +|2201E|INVALID ARGUMENT FOR LOGARITHM|invalid\_argument\_for\_logarithm| +|2201F|INVALID ARGUMENT FOR POWER FUNCTION|invalid\_argument\_for\_power\_function| +|2201G|INVALID ARGUMENT FOR WIDTH BUCKET FUNCTION|invalid\_argument\_for\_width\_bucket\_function| +|22018|INVALID CHARACTER VALUE FOR CAST|invalid\_character\_value\_for\_cast| +|22007|INVALID DATETIME FORMAT|invalid\_datetime\_format| +|22019|INVALID ESCAPE CHARACTER|invalid\_escape\_character| +|2200D|INVALID ESCAPE OCTET|invalid\_escape\_octet| +|22025|INVALID ESCAPE SEQUENCE|invalid\_escape\_sequence| +|22P06|NONSTANDARD USE OF ESCAPE CHARACTER|nonstandard\_use\_of\_escape\_character| +|22010|INVALID INDICATOR PARAMETER VALUE|invalid\_indicator\_parameter\_value| +|22020|INVALID LIMIT VALUE|invalid\_limit\_value| +|22023|INVALID PARAMETER VALUE|invalid\_parameter\_value| +|2201B|INVALID REGULAR EXPRESSION|invalid\_regular\_expression| +|22009|INVALID TIME ZONE DISPLACEMENT VALUE|invalid\_time\_zone\_displacement\_value| +|2200C|INVALID USE OF ESCAPE CHARACTER|invalid\_use\_of\_escape\_character| +|2200G|MOST SPECIFIC TYPE MISMATCH|most\_specific\_type\_mismatch| +|22004|NULL VALUE NOT ALLOWED|null\_value\_not\_allowed| +|22002|NULL VALUE NO INDICATOR PARAMETER|null\_value\_no\_indicator\_parameter| +|22003|NUMERIC VALUE OUT OF RANGE|numeric\_value\_out\_of\_range| +|22026|STRING DATA LENGTH MISMATCH|string\_data\_length\_mismatch| +|22001|STRING DATA RIGHT TRUNCATION|string\_data\_right\_truncation| +|22011|SUBSTRING ERROR|substring\_error| +|22027|TRIM ERROR|trim\_error| +|22024|UNTERMINATED C STRING|unterminated\_c\_string| +|2200F|ZERO LENGTH CHARACTER STRING|zero\_length\_character\_string| +|22P01|FLOATING POINT EXCEPTION|floating\_point\_exception| +|22P02|INVALID TEXT REPRESENTATION|invalid\_text\_representation| +|22P03|INVALID BINARY REPRESENTATION|invalid\_binary\_representation| +|22P04|BAD COPY FILE FORMAT|bad\_copy\_file\_format| +|22P05|UNTRANSLATABLE CHARACTER|untranslatable\_character| +|**Class 23** â Integrity Constraint Violation| +|23000|INTEGRITY CONSTRAINT VIOLATION|integrity\_constraint\_violation| +|23001|RESTRICT VIOLATION|restrict\_violation| +|23502|NOT NULL VIOLATION|not\_null\_violation| +|23503|FOREIGN KEY VIOLATION|foreign\_key\_violation| +|23505|UNIQUE VIOLATION|unique\_violation| +|23514|CHECK VIOLATION|check\_violation| +|**Class 24** â Invalid Cursor State| +|24000|INVALID CURSOR STATE|invalid\_cursor\_state| +|**Class 25** â Invalid Transaction State| +|25000|INVALID TRANSACTION STATE|invalid\_transaction\_state| +|25001|ACTIVE SQL TRANSACTION|active\_sql\_transaction| +|25002|BRANCH TRANSACTION ALREADY ACTIVE|branch\_transaction\_already\_active| +|25008|HELD CURSOR REQUIRES SAME ISOLATION LEVEL|held\_cursor\_requires\_same\_isolation\_level| +|25003|INAPPROPRIATE ACCESS MODE FOR BRANCH TRANSACTION|inappropriate\_access\_mode\_for\_branch\_transaction| +|25004|INAPPROPRIATE ISOLATION LEVEL FOR BRANCH TRANSACTION|inappropriate\_isolation\_level\_for\_branch\_transaction| +|25005|NO ACTIVE SQL TRANSACTION FOR BRANCH TRANSACTION|no\_active\_sql\_transaction\_for\_branch\_transaction| +|25006|READ ONLY SQL TRANSACTION|read\_only\_sql\_transaction| +|25007|SCHEMA AND DATA STATEMENT MIXING NOT SUPPORTED|schema\_and\_data\_statement\_mixing\_not\_supported| +|25P01|NO ACTIVE SQL TRANSACTION|no\_active\_sql\_transaction| +|25P02|IN FAILED SQL TRANSACTION|in\_failed\_sql\_transaction| +|**Class 26** â Invalid SQL Statement Name| +|26000|INVALID SQL STATEMENT NAME|invalid\_sql\_statement\_name| +|**Class 27** â Triggered Data Change Violation| +|27000|TRIGGERED DATA CHANGE VIOLATION|triggered\_data\_change\_violation| +|**Class 28** â Invalid Authorization Specification| +|28000|INVALID AUTHORIZATION SPECIFICATION|invalid\_authorization\_specification| +|**Class 2B** â Dependent Privilege Descriptors Still Exist| +|2B000|DEPENDENT PRIVILEGE DESCRIPTORS STILL EXIST|dependent\_privilege\_descriptors\_still\_exist| +|2BP01|DEPENDENT OBJECTS STILL EXIST|dependent\_objects\_still\_exist| +|**Class 2D** â Invalid Transaction Termination| +|2D000|INVALID TRANSACTION TERMINATION|invalid\_transaction\_termination| +|**Class 2F** â SQL Routine Exception| +|2F000|SQL ROUTINE EXCEPTION|sql\_routine\_exception| +|2F005|FUNCTION EXECUTED NO RETURN STATEMENT|function\_executed\_no\_return\_statement| +|2F002|MODIFYING SQL DATA NOT PERMITTED|modifying\_sql\_data\_not\_permitted| +|2F003|PROHIBITED SQL STATEMENT ATTEMPTED|prohibited\_sql\_statement\_attempted| +|2F004|READING SQL DATA NOT PERMITTED|reading\_sql\_data\_not\_permitted| +|**Class 34** â Invalid Cursor Name| +|34000|INVALID CURSOR NAME|invalid\_cursor\_name| +|**Class 38** â External Routine Exception| +|38000|EXTERNAL ROUTINE EXCEPTION|external\_routine\_exception| +|38001|CONTAINING SQL NOT PERMITTED|containing\_sql\_not\_permitted| +|38002|MODIFYING SQL DATA NOT PERMITTED|modifying\_sql\_data\_not\_permitted| +|38003|PROHIBITED SQL STATEMENT ATTEMPTED|prohibited\_sql\_statement\_attempted| +|38004|READING SQL DATA NOT PERMITTED|reading\_sql\_data\_not\_permitted| +|**Class 39** â External Routine Invocation Exception| +|39000|EXTERNAL ROUTINE INVOCATION EXCEPTION|external\_routine\_invocation\_exception| +|39001|INVALID SQLSTATE RETURNED|invalid\_sqlstate\_returned| +|39004|NULL VALUE NOT ALLOWED|null\_value\_not\_allowed| +|39P01|TRIGGER PROTOCOL VIOLATED|trigger\_protocol\_violated| +|39P02|SRF PROTOCOL VIOLATED|srf\_protocol\_violated| +|**Class 3B** â Savepoint Exception| +|3B000|SAVEPOINT EXCEPTION|savepoint\_exception| +|3B001|INVALID SAVEPOINT SPECIFICATION|invalid\_savepoint\_specification| +|**Class 3D** â Invalid Catalog Name| +|3D000|INVALID CATALOG NAME|invalid\_catalog\_name| +|**Class 3F** â Invalid Schema Name| +|3F000|INVALID SCHEMA NAME|invalid\_schema\_name| +|**Class 40** â Transaction Rollback| +|40000|TRANSACTION ROLLBACK|transaction\_rollback| +|40002|TRANSACTION INTEGRITY CONSTRAINT VIOLATION|transaction\_integrity\_constraint\_violation| +|40001|SERIALIZATION FAILURE|serialization\_failure| +|40003|STATEMENT COMPLETION UNKNOWN|statement\_completion\_unknown| +|40P01|DEADLOCK DETECTED|deadlock\_detected| +|**Class 42** â Syntax Error or Access Rule Violation| +|42000|SYNTAX ERROR OR ACCESS RULE VIOLATION|syntax\_error\_or\_access\_rule\_violation| +|42601|SYNTAX ERROR|syntax\_error| +|42501|INSUFFICIENT PRIVILEGE|insufficient\_privilege| +|42846|CANNOT COERCE|cannot\_coerce| +|42803|GROUPING ERROR|grouping\_error| +|42830|INVALID FOREIGN KEY|invalid\_foreign\_key| +|42602|INVALID NAME|invalid\_name| +|42622|NAME TOO LONG|name\_too\_long| +|42939|RESERVED NAME|reserved\_name| +|42804|DATATYPE MISMATCH|datatype\_mismatch| +|42P18|INDETERMINATE DATATYPE|indeterminate\_datatype| +|42809|WRONG OBJECT TYPE|wrong\_object\_type| +|42703|UNDEFINED COLUMN|undefined\_column| +|42883|UNDEFINED FUNCTION|undefined\_function| +|42P01|UNDEFINED TABLE|undefined\_table| +|42P02|UNDEFINED PARAMETER|undefined\_parameter| +|42704|UNDEFINED OBJECT|undefined\_object| +|42701|DUPLICATE COLUMN|duplicate\_column| +|42P03|DUPLICATE CURSOR|duplicate\_cursor| +|42P04|DUPLICATE DATABASE|duplicate\_database| +|42723|DUPLICATE FUNCTION|duplicate\_function| +|42P05|DUPLICATE PREPARED STATEMENT|duplicate\_prepared\_statement| +|42P06|DUPLICATE SCHEMA|duplicate\_schema| +|42P07|DUPLICATE TABLE|duplicate\_table| +|42712|DUPLICATE ALIAS|duplicate\_alias| +|42710|DUPLICATE OBJECT|duplicate\_object| +|42702|AMBIGUOUS COLUMN|ambiguous\_column| +|42725|AMBIGUOUS FUNCTION|ambiguous\_function| +|42P08|AMBIGUOUS PARAMETER|ambiguous\_parameter| +|42P09|AMBIGUOUS ALIAS|ambiguous\_alias| +|42P10|INVALID COLUMN REFERENCE|invalid\_column\_reference| +|42611|INVALID COLUMN DEFINITION|invalid\_column\_definition| +|42P11|INVALID CURSOR DEFINITION|invalid\_cursor\_definition| +|42P12|INVALID DATABASE DEFINITION|invalid\_database\_definition| +|42P13|INVALID FUNCTION DEFINITION|invalid\_function\_definition| +|42P14|INVALID PREPARED STATEMENT DEFINITION|invalid\_prepared\_statement\_definition| +|42P15|INVALID SCHEMA DEFINITION|invalid\_schema\_definition| +|42P16|INVALID TABLE DEFINITION|invalid\_table\_definition| +|42P17|INVALID OBJECT DEFINITION|invalid\_object\_definition| +|**Class 44** â WITH CHECK OPTION Violation| +|44000|WITH CHECK OPTION VIOLATION|with\_check\_option\_violation| +|**Class 53** â Insufficient Resources| +|53000|INSUFFICIENT RESOURCES|insufficient\_resources| +|53100|DISK FULL|disk\_full| +|53200|OUT OF MEMORY|out\_of\_memory| +|53300|TOO MANY CONNECTIONS|too\_many\_connections| +|**Class 54** â Program Limit Exceeded| +|54000|PROGRAM LIMIT EXCEEDED|program\_limit\_exceeded| +|54001|STATEMENT TOO COMPLEX|statement\_too\_complex| +|54011|TOO MANY COLUMNS|too\_many\_columns| +|54023|TOO MANY ARGUMENTS|too\_many\_arguments| +|**Class 55** â Object Not In Prerequisite State| +|55000|OBJECT NOT IN PREREQUISITE STATE|object\_not\_in\_prerequisite\_state| +|55006|OBJECT IN USE|object\_in\_use| +|55P02|CANT CHANGE RUNTIME PARAM|cant\_change\_runtime\_param| +|55P03|LOCK NOT AVAILABLE|lock\_not\_available| +|**Class 57** â Operator Intervention| +|57000|OPERATOR INTERVENTION|operator\_intervention| +|57014|QUERY CANCELED|query\_canceled| +|57P01|ADMIN SHUTDOWN|admin\_shutdown| +|57P02|CRASH SHUTDOWN|crash\_shutdown| +|57P03|CANNOT CONNECT NOW|cannot\_connect\_now| +|**Class 58** â System Error \(errors external to HAWQ \)| +|58030|IO ERROR|io\_error| +|58P01|UNDEFINED FILE|undefined\_file| +|58P02|DUPLICATE FILE|duplicate\_file| +|Class F0 â Configuration File Error| +|F0000|CONFIG FILE ERROR|config\_file\_error| +|F0001|LOCK FILE EXISTS|lock\_file\_exists| +|**Class P0** â PL/pgSQL Error| +|P0000|PLPGSQL ERROR|plpgsql\_error| +|P0001|RAISE EXCEPTION|raise\_exception| +|P0002|NO DATA FOUND|no\_data\_found| +|P0003|TOO MANY ROWS|too\_many\_rows| +|**Class XX** â Internal Error| +|XX000|INTERNAL ERROR|internal\_error| +|XX001|DATA CORRUPTED|data\_corrupted| +|XX002|INDEX CORRUPTED|index\_corrupted| http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/markdown/admin/setuphawqopenv.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/admin/setuphawqopenv.html.md.erb b/markdown/admin/setuphawqopenv.html.md.erb new file mode 100644 index 0000000..9d9b731 --- /dev/null +++ b/markdown/admin/setuphawqopenv.html.md.erb @@ -0,0 +1,81 @@ +--- +title: Introducing the HAWQ Operating Environment +--- + +Before invoking operations on a HAWQ cluster, you must set up your HAWQ environment. This set up is required for both administrative and non-administrative HAWQ users. + +## <a id="hawq_setupenv"></a>Procedure: Setting Up Your HAWQ Operating Environment + +HAWQ installs a script that you can use to set up your HAWQ cluster environment. The `greenplum_path.sh` script, located in your HAWQ root install directory, sets `$PATH` and other environment variables to find HAWQ files. Most importantly, `greenplum_path.sh` sets the `$GPHOME` environment variable to point to the root directory of the HAWQ installation. If you installed HAWQ from a product distribution, the HAWQ root is typically `/usr/local/hawq`. If you built HAWQ from source or downloaded the tarball, you will have selected an install root directory on your own. + +Perform the following steps to set up your HAWQ operating environment: + +1. Log in to the HAWQ node as the desired user. For example: + + ``` shell + $ ssh gpadmin@<master> + gpadmin@master$ + ``` + + Or, if you are already logged in to \<node\-type\> as a different user, switch to the desired user. For example: + + ``` shell + gpadmin@master$ su - <hawq-user> + Password: + hawq-user@master$ + ``` + +2. Set up your HAWQ operating environment by sourcing the `greenplum_path.sh` file: + + ``` shell + hawq-node$ source /usr/local/hawq/greenplum_path.sh + ``` + + If you built HAWQ from source or downloaded the tarball, substitute the path to the installed or extracted `greenplum_path.sh` file \(for example `/opt/hawq-2.1.0.0/greenplum_path.sh`\). + + +3. Edit your `.bash_profile` or other shell initialization file to source `greenplum_path.sh` on login. For example, add: + + ``` shell + source /usr/local/hawq/greenplum_path.sh + ``` + +4. Set HAWQ-specific environment variables relevant to your deployment in your shell initialization file. These include `PGAPPNAME`, `PGDATABASE`, `PGHOST`, `PGPORT`, and `PGUSER.` For example: + + 1. If you use a custom HAWQ master port number, make this port number the default by setting the `PGPORT` environment variable in your shell initialization file; add: + + ``` shell + export PGPORT=10432 + ``` + + Setting `PGPORT` simplifies `psql` invocation by providing a default for the `-p` (port) option. + + 1. If you will routinely operate on a specific database, make this database the default by setting the `PGDATABASE` environment variable in your shell initialization file: + + ``` shell + export PGDATABASE=<database-name> + ``` + + Setting `PGDATABASE` simplifies `psql` invocation by providing a default for the `-d` (database) option. + + You may choose to set additional HAWQ deployment-specific environment variables. See [Environment Variables](../reference/HAWQEnvironmentVariables.html#optionalenvironmentvariables). + +## <a id="hawq_env_files_and_dirs"></a>HAWQ Files and Directories + +The following table identifies some files and directories of interest in a default HAWQ installation. Unless otherwise specified, the table entries are relative to `$GPHOME`. + +|File/Directory | Contents | +|---------------------------------|---------------------| +| $HOME/hawqAdminLogs/ | Default HAWQ management utility log file directory | +| greenplum_path.sh | HAWQ environment set-up script | +| bin/ | HAWQ admin, client, database, and administration utilities | +| etc/ | HAWQ configuration files, including `hawq-site.xml` | +| include/ | HDFS, PostgreSQL, `libpq` header files | +| lib/ | HAWQ libraries | +| lib/postgresql/ | PostgreSQL shared libraries and JAR files | +| share/postgresql/ | PostgreSQL and procedural languages samples and scripts | +| /data/hawq/[master|segment]/ | Default location of HAWQ master and segment data directories | +| /data/hawq/[master|segment]/pg_log/ | Default location of HAWQ master and segment log file directories | +| /etc/pxf/conf/ | PXF service and configuration files | +| /usr/lib/pxf/ | PXF service and plug-in shared libraries | +| /usr/hdp/current/ | HDP runtime and configuration files |
