Repository: incubator-hawq-docs Updated Branches: refs/heads/develop 42235b15f -> a362cb696
HAWQ-1192 - add discussion of heap and thread config for pxf (This closes #69) Squashed commit of the following: commit a289bef8419033a7da7b502375bd881cc54a8fb7 Author: Lisa Owen <[email protected]> Date: Fri Dec 9 15:16:07 2016 -0800 change title of pxf memory issue section commit 06718c7f740a516a92673e4ba3a0d379ca11a053 Author: Lisa Owen <[email protected]> Date: Tue Dec 6 11:55:53 2016 -0800 add xref to new pxf heap/thread config section commit b67e9edc53171b2007d4513b0cc851055953f8bb Author: Lisa Owen <[email protected]> Date: Tue Dec 6 11:13:35 2016 -0800 add discussion of heap and thread config for pxf Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/a362cb69 Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/a362cb69 Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/a362cb69 Branch: refs/heads/develop Commit: a362cb696235ffd740c553abc8e50db777e37784 Parents: 42235b1 Author: David Yozie <[email protected]> Authored: Mon Dec 12 15:06:10 2016 -0800 Committer: David Yozie <[email protected]> Committed: Mon Dec 12 15:06:10 2016 -0800 ---------------------------------------------------------------------- pxf/ConfigurePXF.html.md.erb | 2 + pxf/TroubleshootingPXF.html.md.erb | 105 +++++++++++++++++++++++++++++++- 2 files changed, 106 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a362cb69/pxf/ConfigurePXF.html.md.erb ---------------------------------------------------------------------- diff --git a/pxf/ConfigurePXF.html.md.erb b/pxf/ConfigurePXF.html.md.erb index 087a89a..fec6b27 100644 --- a/pxf/ConfigurePXF.html.md.erb +++ b/pxf/ConfigurePXF.html.md.erb @@ -27,6 +27,8 @@ JVM_OPTS="-Xmx512M -Xss256K" After adding or modifying the JVM command line options, the PXF service must be restarted. +(Refer to [Addressing PXF Memory Issues](TroubleshootingPXF.html#pxf-memcfg) for a related discussion of the configuration options available to address memory issues in your PXF deployment.) + ## <a id="topic_i3f_hvm_ss"></a>Using PXF on a Secure HDFS Cluster You can use PXF on a secure HDFS cluster. Read, write, and analyze operations for PXF tables on HDFS files are enabled. It requires no changes to preexisting PXF tables from a previous version. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a362cb69/pxf/TroubleshootingPXF.html.md.erb ---------------------------------------------------------------------- diff --git a/pxf/TroubleshootingPXF.html.md.erb b/pxf/TroubleshootingPXF.html.md.erb index f38b3c8..1f814de 100644 --- a/pxf/TroubleshootingPXF.html.md.erb +++ b/pxf/TroubleshootingPXF.html.md.erb @@ -200,4 +200,107 @@ Examine/collect the log messages from `stdout`. ``` sql gpadmin=# SET client_min_messages=NOTICE -``` \ No newline at end of file +``` + + +## <a id="pxf-memcfg"></a>Addressing PXF Memory Issues + +The Java heap size can be a limiting factor in PXFâs ability to serve many concurrent requests or to run queries against large tables. + +You may run into situations where a query will hang or fail with an Out of Memory exception (OOM). This typically occurs when many threads are reading different data fragments from an external table and insufficient heap space exists to open all fragments at the same time. To avert or remedy this situation, Pivotal recommends first increasing the Java maximum heap size or decreasing the Tomcat maximum number of threads, depending upon what works best for your system configuration. + +**Note**: The configuration changes described in this topic require modifying config files on *each* PXF node in your HAWQ cluster. After performing the updates, be sure to verify that the configuration on all PXF nodes is the same. + +### <a id="pxf-heapcfg"></a>Increasing the Maximum Heap Size + +Each PXF node is configured with a default Java heap size of 512MB. If the nodes in your cluster have an ample amount of memory, increasing the amount allocated to the PXF agents is the best approach. Pivotal recommends a heap size value between 1-2GB. + +The steps you perform to increase the PXF agent heap size will differ depending upon whether or not the PXF service is currently running. **You must perform the configuration changes on each PXF node in your HAWQ cluster.** + +If the PXF service has not yet been started, increase the heap size in the PXF Tomcat `setenv.sh` template: + +1. Open the `/opt/pxf/tomcat-templates/bin/setenv.sh` file in a text editor. + + ``` shell + root@pxf-node$ vi /opt/pxf/tomcat-templates/bin/setenv.sh + ``` + +2. Update the `-Xmx` option to the desired value in the `JVM_OPTS` setting: + + ``` shell + JVM_OPTS="-Xmx1024M -Xss256K" + ``` + +If the PXF service is running, increase the heap size directly in the PXF `setenv.sh` script: + +1. Open `/var/pxf/pxf-service/bin/setenv.sh` in a text editor. + + ``` shell + root@pxf-node$ vi /var/pxf/pxf-service/bin/setenv.sh + ``` + +2. Update the `-Xmx` option to the desired value in the `JVM_OPTS` setting: + + ``` shell + JVM_OPTS="-Xmx1024M -Xss256K" + ``` + +3. Restart PXF: + + 1. If you use Ambari to manage your cluster, restart the PXF service via the Ambari console. + 2. If you do not use Ambari, restart the PXF service from the command line on each node: + + ``` shell + root@pxf-node$ service pxf-service restart + ``` + +### <a id="pxf-heapcfg"></a>Decreasing the Maximum Number of Threads + +If increasing the maximum heap size is not suitable for your HAWQ cluster, try decreasing the number of concurrent working threads configured for the underlying Tomcat web application. A decrease in the number of running threads will prevent any PXF node from exhausting its memory, while ensuring that current queries run to completion (albeit a bit slower). As Tomcat's default behavior is to queue requests until a thread is free, decreasing this value will not result in denied requests. + +The Tomcat default maximum number of threads is 300. Pivotal recommends decreasing the maximum number of threads to under 6. (If you plan to run large workloads on a large number of files using a Hive profile, Pivotal recommends you pick an even lower value.) + +The steps to decrease the maximum number of Tomcat threads will differ depending upon whether or not the PXF service is currently running. **You must perform the configuration changes on each PXF node in your HAWQ cluster.** + +If the PXF service has not yet been started, decrease the maximum number of threads via the PXF Tomcat `server.xml` template: + +1. Open the `/opt/pxf/tomcat-templates/conf/server.xml` file in a text editor. + + ``` shell + root@pxf-node$ vi /opt/pxf/tomcat-templates/conf/server.xml + ``` + +2. Update the `Catalina` `Executor` block to identify the desired `maxThreads` value: + + ``` xml + <Executor maxThreads="2" + minSpareThreads="50" + name="tomcatThreadPool" + namePrefix="tomcat-http--"/> + ``` + +If the PXF service is running, decrease the maximum number of Tomcat threads directly in the PXF `server.xml` file: + +1. Open the `/var/pxf/pxf-service/conf/server.xml` file in a text editor. + + ``` shell + root@pxf-node$ vi /var/pxf/pxf-service/conf/server.xml + ``` + +2. Update the `Catalina` `Executor` block to identify the desired `maxThreads` value: + + ``` xml + <Executor maxThreads="2" + minSpareThreads="50" + name="tomcatThreadPool" + namePrefix="tomcat-http--"/> + ``` + +3. Restart PXF: + + 1. If you use Ambari to manage your cluster, restart the PXF service via the Ambari console. + 2. If you do not use Ambari, restart the PXF service from the command line on each node: + + ``` shell + root@pxf-node$ service pxf-service restart + ```
