[GitHub] incubator-hawq-docs pull request #135: HAWQ-1368 Add option to hawq register...
Github user lisakowen commented on a diff in the pull request:
https://github.com/apache/incubator-hawq-docs/pull/135#discussion_r164250784
--- Diff: markdown/reference/cli/admin_utilities/hawqregister.html.md.erb
---
@@ -200,6 +200,8 @@ group {
-\\\-version
Show the version of this utility, then exit.
+-l, -\\\-logdir \<logfile\_directory\>
+Specifies the log directory that `hawq register` uses for logs. The
default is `~/hawqAdminLogs/`.
--- End diff --
same comment as hawq extract ... how about "Specifies the directory that
`hawq register` uses for log files."?
---
[GitHub] incubator-hawq-docs pull request #134: HAWQ-1553 Add option to hawq extract ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/134#discussion_r164250622 --- Diff: markdown/reference/cli/admin_utilities/hawqextract.html.md.erb --- @@ -73,6 +73,9 @@ where: -\\\-version Displays the version of this utility. +-l, -\\\-logdir \<logfile\_directory\> +Specifies the log directory that `hawq extract` uses for logs. The default is `~/hawqAdminLogs/`. --- End diff -- how about "Specifies the directory that `hawq extract` uses for log files."? ---
[GitHub] incubator-hawq-docs pull request #133: server_ticket_renew_interval guc, YAR...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/133 server_ticket_renew_interval guc, YARN HA props - document server_ticket_renew_interval GUC - clarify YARN HA property lists links: - http://docs-hdb-review.cfapps.io/review/hawq/resourcemgmt/YARNIntegration.html#highlyavailableyarn - added new paragraph to end of the section to identify master and standby first in property lists - http://docs-hdb-review.cfapps.io/review/hawq/reference/guc/parameter_definitions.html#server_ticket_renew_interval - add new server_ticket_renew_interval GUC to ref guide - http://docs-hdb-review.cfapps.io/review/hawq/clientaccess/kerberos-securehdfs.html#server_ticket_renew - add section documenting the guc and how to change You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/kerbticket-yarnha Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #133 commit 13b29193ca057eab6d70e25ed09c60f12bb56d59 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-10-26T22:04:57Z server_ticket_renew_interval guc, YARN HA props ---
[GitHub] incubator-hawq-docs pull request #132: add support for active directory KDC ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/132#discussion_r147250402 --- Diff: markdown/clientaccess/kerberos-userauth.html.md.erb --- @@ -0,0 +1,459 @@ +--- +title: Configuring Kerberos User Authentication for HAWQ +--- + + + +When Kerberos authentication is enabled at the user level, HAWQ uses the Generic Security Service Application Program Interface \(GSSAPI\) to provide automatic authentication \(single sign-on\). When HAWQ uses Kerberos user authentication, HAWQ itself and the HAWQ users \(roles\) that require Kerberos authentication require a principal and keytab. When a user attempts to log in to HAWQ, HAWQ uses its Kerberos principal to connect to the Kerberos server, and presents the user's principal for Kerberos validation. If the user principal is valid, login succeeds and the user can access HAWQ. Conversely, the login fails and HAWQ denies access to the user if the principal is not valid. + +When HAWQ utilizes Kerberos for user authentication, it uses a standard principal to connect to the Kerberos KDC. The format of this principal is `postgres/@`, where \<FQDN\_of\_master\> refers to the fully qualified distinguish name of the HAWQ master node. + +(You may choose to configure HAWQ user principals before you enable Kerberos user authentication for HAWQ. See [Configuring Kerberos-Authenticated HAWQ Users](#hawq_kerb_user_cfg).) + +The procedure to configure Kerberos user authentication for HAWQ includes: + +If you use an MIT Kerberos KDC Server: +**Step 1a**: [Configuring the HAWQ Principals using an MIT KDC Server](#hawq_kerb_cfg_mitkdc) + +If you use an Active Directory Kerberos KDC Server: +**Step 1b**: [Configuring the HAWQ Principal using an AD KDC Server](#hawq_kerb_cfg_adkdc) + +**Step 2**: [Configuring HAWQ to use Kerberos Authentication](#hawq_kerb_cfg) +**Step 3**: [Configuring Kerberos-Authenticated HAWQ Users](#hawq_kerb_user_cfg) +**Step 4**: [Authenticating User Access to HAWQ](#hawq_kerb_dbaccess) + +## Step 1a: Configuring the HAWQ Principals using an MIT KDC Server + +Perform the following procedure to configure HAWQ Kerberos and `gpadmin` principals when you are using an MIT KDC server. + +**Note**: Some operations may differ based on whether or not you have configured secure HDFS. These operations are called out below. + +1. Log in to the Kerberos KDC server system: + +``` shell +$ ssh root@ +root@kdc-server$ +``` + +2. Create a keytab entry for the HAWQ `postgres/` principal using the `kadmin.local` command. Substitute the HAWQ master node fully qualified distinguished hostname and your Kerberos realm. For example: + +``` shell +root@kdc-server$ kadmin.local -q "addprinc -randkey postgres/@REALM.DOMAIN" +``` + +The `addprinc` command adds the principal `postgres/` to the KDC managing your \<realm\>. + +3. Generate a keytab file for the HAWQ `postgres/` principal. Provide the same name you used to create the principal. + +**If you have configured Kerberos for your HDFS filesystem**, add the keytab to the HAWQ client HDFS keytab file: + +``` shell +root@kdc-server$ kadmin.local -q "xst -norandkey -k /etc/security/keytabs/hawq.service.keytab postgres/@REALM.DOMAIN" +``` + +**Otherwise**, generate a new file for the keytab: + +``` shell +root@kdc-server$ kadmin.local -q "xst -norandkey -k hawq-krb5.keytab postgres/@REALM.DOMAIN" +``` + +4. Use the `klist` command to view the key you just generated: + +``` shell +root@kdc-server$ klist -ket ./hawq-krb5.keytab +``` + +Or: + +``` shell +root@kdc-server$ klist -ket /etc/security/keytabs/hawq.service.keytab +``` + +The `-ket` option lists the keytabs and encryption types in the identified key file. + +5. When you enable Kerberos user authentication for HAWQ, you must create a Kerberos principal for `gpadmin` or another HAWQ administrative user. Create a Kerberos principal for the HAWQ `gpadmin` administrative role, substituting your Kerberos realm. For example: + +``` shell +root@kdc-server$ kadmin.local -q "addprinc -pw changeme gpadmin@REALM.DOMAIN" +``` + +This `addprinc` command adds the principal `gpadmin` to the Kerberos KDC managing your \<realm\>. When you invoke `kadmin.local` as specified in the example above, `gpadmin` will be required to provide the password identified by the `-pw` option when authenticating. Alternatively, you can create a keytab file for the `gpadm
[GitHub] incubator-hawq-docs pull request #132: add support for active directory KDC ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/132 add support for active directory KDC server restructured the kerberos topic - broke it up into 3 pages. link to the "landing" page: - http://docs-hdb-review.cfapps.io/review/hawq/clientaccess/kerberos.html you can click the > in the left subnav to see the subtopics. the bulk of the AD KDC content is in these links: - http://docs-hdb-review.cfapps.io/review/hawq/clientaccess/kerberos.html#kerberos_prereq - http://docs-hdb-review.cfapps.io/review/hawq/clientaccess/kerberos-userauth.html#hawq_kerb_cfg_adkdc - http://docs-hdb-review.cfapps.io/review/hawq/clientaccess/kerberos-userauth.html#hawq_kerb_user_cfg You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/kerberos-ad-kdc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/132.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #132 commit 533e02021af115ae21d0b584f8936e7aec0d6793 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-10-24T23:25:16Z add support for active directory KDC server ---
[GitHub] incubator-hawq-docs pull request #123: HAWQ-1480 - Added documentation for p...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/123#discussion_r143597015 --- Diff: book/master_middleman/source/subnavs/apache-hawq-nav.erb --- @@ -117,6 +117,9 @@ HAWQ Administrative Log Files + +HAWQ packcore utility --- End diff -- this is very minor - how about capitalizing the U in "utility"? ---
[GitHub] incubator-hawq-docs pull request #128: HAWQ-1479 - doc ranger kerberos integ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/128#discussion_r127225477 --- Diff: markdown/ranger/ranger-kerberos.html.md.erb --- @@ -0,0 +1,209 @@ +--- +title: HAWQ Ranger Kerberos Integration +--- + + + +When you have enabled Ranger Authorization for HAWQ, your HAWQ installation includes the Ranger Administrative UI and HAWQ Ranger Plug-in Service. + +Specific HAWQ Ranger configuration is required when Kerberos authentication is enabled for HAWQ or for Ranger. You must configure Kerberos support for: + +- HAWQ resource lookup by the Ranger Administration host during HAWQ policy definition +- HAWQ Ranger Plug-in Service communication with the Ranger Administration host for policy refresh + +Use the following procedures to configure Kerberos support for your Ranger-authorized HAWQ cluster. + +## Prerequisites + +Before you configure Kerberos for your Ranger-authorized HAWQ cluster, ensure that you have: + +- Installed Java 1.7.0\_17 or later on all nodes in your cluster. Java 1.7.0_17 is required to use Kerberos-authenticated JDBC on Red Hat Enterprise Linux 6.x or 7.x. +- (Non-OpenJDK Java installations) Installed the Java Cryptography Extension (JCE) on all nodes in your cluster. +- If you manage your cluster with Ambari, you installed the JCE on each node before you enabled Kerberos with the Ambari **Kerberos Security Wizard**. +- If you manage your cluster from the command line, you must manually install the extension on these systems. +- Noted the host name or IP address of your Ranger Administration host (\<ranger-admin-node\>) and HAWQ master (\<master\>) nodes. +- Identified an existing Kerberos Key Distribution Center (KDC) or set up your KDC as described in [Install and Configure a Kerberos KDC Server](../clientaccess/kerberos.html#task_setup_kdc). +- Note the host name or IP address of your KDC (\<kdc-server\>). +- Note the name of the Kerberos \<realm\> in which your cluster resides. +- Enabled Ranger Authorization for HAWQ. See [Configuring HAWQ to use Ranger Policy Management](ranger-integration-config.html). + + +## Configure Ranger for Kerberized HAWQ + +When you define HAWQ Ranger authorization policies, the Ranger Administration Host uses JDBC to connect to HAWQ during policy definition to look up policy resource names. When Kerberos user authentication is enabled for HAWQ, you must configure this connection for Kerberos. + +To configure Ranger access to a HAWQ cluster enabled with Kerberos user authentication, you must: + +- Identify an existing HAWQ administrative role or create a new HAWQ administrative role for Ranger lookup of HAWQ resources +- Create a Kerberos principal for the lookup role +- Update the Ranger HAWQ service definition + +### Procedure + +Perform the following procedure to enable the Ranger Administration Host to look up resources in your kerberized HAWQ cluster. You will perform operations on the HAWQ \<master\>, \<ranger-admin-node\>, and \<kdc-server\> nodes. + +1. Log in to the HAWQ master node and set up your environment: + +``` shell +$ ssh gpadmin@ +gpadmin@master$ . /usr/local/hawq/greenplum_path.sh +``` + +2. Identify an existing HAWQ administrative role or create a new HAWQ administrative role for Ranger resource lookup. For example, to create a new administrative role: + +``` shell +gpadmin@master$ psql -c 'CREATE ROLE "rangerlookup_hawq" with LOGIN SUPERUSER;' +``` + +You may choose a different name for the Ranger lookup role. + +3. Log in to the KDC server system and generate a principal for the HAWQ `rangerlookup_hawq` role. Substitute your Kerberos \<realm\>. For example: + +``` shell +$ ssh root@ +root@kdc-server$ kadmin.local -q "addprinc -pw changeme rangerlookup_hawq@REALM.DOMAIN" +``` + +You do not need to generate a keytab file for the `rangerlookup_hawq` principal because you will provide the password in the HAWQ service definition of the Ranger Admin UI. + +4. Start the Ranger Admin UI in a supported web browser. The default URL is \<ranger-admin-node\>:6080. + +5. Locate the HAWQ service definition and click the **Edit** button. + +6. Update the applicable **Config Properties** fields: + +**HAWQ User Name*** : Enter the HAWQ Ranger lookup role you identified or created in Step 2 above. +**HAWQ User Password*** : Enter the password you assigned to the HAWQ Ranger lookup role principal. +**HAWQ Kerberos Service Name** : Enter `postgres` or your cus
[GitHub] incubator-hawq-docs pull request #127: HAWQ-1497 - kerberos docs refactoring
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/127 HAWQ-1497 - kerberos docs refactoring restructured the general enable/disable kerberos docs to be more modular - now distinguishing between secure hdfs config and config'ing for kerberos user authentication for hawq - new section on configuring hawq/pxf for secure hdfs - new section disabling kerberos user authentication for hawq - new section on kerberos considerations for clients residing on non-hawq nodes - general clarifications of content You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/kerberos-refactor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #127 commit 751d96aa9af133ee42004079347320fdc057c1fb Author: Lisa Owen <lo...@pivotal.io> Date: 2017-07-05T21:55:49Z HAWQ-1497 - kerberos docs refactoring - restructured the general enable/disable kerberos docs to be more modular - now distinguishing between secure hdfs config and config'ing for kerberos user authentication for hawq - new section on configuring hawq/pxf for secure hdfs - new section disabling kerberos user authentication for hawq - new section on kerberos considerations for clients residing on non-hawq nodes - general clarifications of content --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #126: HAWQ-1491 - create usage docs for Hiv...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/126 HAWQ-1491 - create usage docs for HiveVectorizedORC profile update hawq docs for new HiveVectorizedORC profile. - add example to hive plug-in page - include the profile and accessor/fragmenter/resolver classes in the appropriate tables in the other docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/pxf-hivevectorizedorc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/126.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #126 commit dfd692cb355e9505e82669c311c413c06ae8518e Author: Lisa Owen <lo...@pivotal.io> Date: 2017-06-27T19:56:45Z HAWQ-1491 - create usage docs for HiveVectorizedORC profile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #125: clarify pxf writable external tables
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/125 clarify pxf writable external tables - HBase profile does not support writable external tables - add note to general pxf external tables discussion You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/pxfwet-clarify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/125.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #125 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #124: HAWQ-1435 document new pxf jdbc plug-...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/124 HAWQ-1435 document new pxf jdbc plug-in document the community-contributed PXF JDBC plug-in. include a simple mysql example. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/pxf-jdbc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/124.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #124 commit a008125b2864c3acbc3b630030cb614a5ea2679f Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-19T00:13:57Z document new pxf jdbc plug-in --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #122: analyze not supported on json, hbase ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/122 analyze not supported on json, hbase external tables call out that analyze is not supported on external tables created with the json or hbase profiles. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/analyze-ext-tbl Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/122.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #122 commit 9eeb5b91ffa871d0176dba9e7935aabec7d93c20 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-05-31T00:25:44Z analyze not supported on json, hbase external tables --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #121: HAWQ-1474 - document minimal psql cli...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/121 HAWQ-1474 - document minimal psql client pkg create/install/run document how to create a minimal psql client "package" including instructions on installing and using the package. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/hawq-psql-client Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/121.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #121 commit d8a2c5f4ff1bc3c56a239f90e3fb02dd577c73af Author: Lisa Owen <lo...@pivotal.io> Date: 2017-05-23T22:33:33Z document minimal psql client pkg create/install/run --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #120: document ranger plugin service high a...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/120 document ranger plugin service high availability hawq/ranger HA info, including: - adding subnav entry for HA - add option step to configure standby RPS - document new hawq_rps_check_local_interval guc - update cli and ambari "activating standby" procedures You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/rps-ha Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/120.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #120 commit 4e3c775928c04c90e0a526d7acaed4d8e90c9ef3 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-05-18T22:58:26Z document ranger plugin service high availability --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #98: HAWQ-1296 - initial draft of hawq gett...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/98 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #117: HAWQ-1424 - new alter sequence ref pa...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/117 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #117: new alter sequence ref page
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/117 new alter sequence ref page add ref page for "alter sequence": - update create/drop sequence see also to include new alter sequence page - fix some formatting of superscript, functions used info from hawq "create sequence" page and postgres "alter sequence" page. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/alter-sequence Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit 23eb9dcd25ab139f82f9633df90831cda2240150 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-05T01:03:30Z new alter sequence ref page --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #116: add ranger-related gucs - hawq_acl_ty...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/116 add ranger-related gucs - hawq_acl_type, hawq_rps_address_port did not create a JIRA for this. added hawq_acl_type and hawq_rps_address port to reference docs. created a new guc category for them - "Ranger Configuration Parameters". You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ranger-gucs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/116.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #116 commit 426af7b76528a631b02ef9f84acf0aac70925208 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-05T22:19:32Z add ranger-related gucs - hawq_acl_type, hawq_rps_address_port --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #115: miscellaneous doc updates
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/115 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #115: miscellaneous doc updates
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/115 miscellaneous doc updates did not create a JIRA for these miscellaneous doc updates: - add missing apache header to ambari-admin and ambari-rest-api files - consolidate images and mdimages directories - consolidated to images/ ** created a new directory, images/unused, for all of the unused images. includes some ambari screenshots and other misc diagrams. let me know if i should just remove them. - remove references to dfs_url where appropriate You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/misc-updates-lo11 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/115.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #115 commit decca2ae91828610e6f0ea8239cff551f74509a1 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-04T21:23:04Z miscellaneous doc updates - add missing apache header - consolidate images directories - remove references to dfs_url where appropriate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #114: restructure example scenario
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/114 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #114: restructure example scenario
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/114 restructure example scenario removed the steps, other. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ranger-scenario Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #114 commit 6fa6871fb4c797f459d464861f8b9212ce476d0a Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-04T18:14:55Z restructure example scenario --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #113: misc ranger doc updates
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/113 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #113: misc ranger doc updates
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/113 misc ranger doc updates - correct create untrusted function perms/resources - clarify schema-level sql cmds - clarify create table in tablespace; reformat - fix link to superuser checks section You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/policy-misc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #113 commit f60070c60c206ae3a6b225ccb4ab747a88c16430 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-04T16:10:38Z misc ranger updates: - correct create untrusted function perms/resources - clarify schema-level sql cmds - clarify create table in tablespace; reformat - fix link to superuser checks section --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #112: reformat sql cmd/perm/resource table ...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/112 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #110: add a section on ranger integration s...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/110 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #111: policy topic restructure
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/111 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #112: reformat sql cmd/perm/resource table ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/112 reformat sql cmd/perm/resource table using html title says it all. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/sqlcmds-reformat Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit b81aa6ba1431c2436fa91728df26754f5c966ed5 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-03T23:21:27Z reformat sql cmd/perm/resource table using html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #111: policy topic restructure
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/111 policy topic restructure restructure the policy topic a bit. pull out sql command summary and permissions supported per resource to new pages. move troubleshooting to integ/config page. added example scenario and screenshots. added resource/privilege summary table. misc other things. the subnav changes are not part of this PR. PENDING - reformatting of the sql command summary table. you should be able to review the other components. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/policy-rework Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #111 commit 53339df70f223ee94448cf55194937ebb0b02678 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-04-03T16:10:39Z policy topic restructure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #110: add a section on ranger integration s...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/110 add a section on ranger integration status add a section identifying how the user can determine the status of ranger integration in their cluster. i added this to the ranger-specific doc. not sure if some or all of this info should be relocated elsewhere in the doc set. (maybe after beta?). You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ranger-integstatus Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/110.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #110 commit b4a4f61bc3983679bafe8c2f728025cf8ebf106e Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-31T17:55:03Z add a section on ranger integration status --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #109: deferring super-user events discussio...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/109#discussion_r109073809 --- Diff: markdown/ranger/ranger-policy-creation.html.md.erb --- @@ -60,6 +60,8 @@ HAWQ *always* employs its native authorization for operations on its catalog. HA - `CREATE`, `DROP`, or `ALTER` commands for resource queues - `CREATE ROLE`, `DROP ROLE`, `SET ROLE`, `createuser`, `dropuser` - `CREATE TABLESPACE`, `DROP TABLESPACE` (Ranger does manage authorization for creating tables and indexes _within_ an existing tablespace.) +- Built-in functions such as pg\_logdir\_ls, pg\_ls\_dir, pg\_read\_file, pg\_reload\_conf, pg\_rotate\_logfile, pg\_signal\_backend, pg\_start\_backup, pg\_stat\_file, pg\_stat\_get\_activity, pg\_stat\_get\_backend\_activity\_start, pg\_stat\_get\_backend\_activity, pg\_stat\_get\_backend\_client\_addr, pg\_stat\_get\_backend\_client\_port, pg\_stat\_get\_backend\_start, pg\_stat\_get\_backend\_waiting, pg\_stop\_backup, pg\_switch\_xlog, and pg\_stat\_reset. --- End diff -- the second bullet was a catch-all for catalog functions. i should we remove this bullet, or keep it in to cover non-listed catalog functions? (i am not familiar with all of the catalog functions. ) looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #108: add config section, misc
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/108 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #108: add config section, misc
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/108#discussion_r109044924 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -124,8 +116,39 @@ Once the connection between HAWQ and Ranger is configured, you can either set up 4. Click **Add Property...** and add the new property, `hawq_acl_type=ranger` property. (If the property already exists, change its value from `standalone` (the default) to `ranger`.) 5. Click **Save** to save your changes. 6. Select **Service Actions > Restart All** and confirm that you want to restart the HAWQ cluster. - -## Changing the Frequency of Policy Caching + +## Custom Configuration + +Configuration files for the HAWQ Ranger Plug-in Service are located in the `$GPHOME/ranger/etc` directory. These files include: + +| File | Description | +|-|---| +| ranger-hawq-audit.xml | HAWQ Ranger audit-related configuration, including the audit provider (log4j, Solr, HDFS) and provider-specific configuration | +| ranger-hawq-security.xml | HAWQ Ranger service configuration, including the policy change polling interval | +| rps.properties | HAWQ Ranger deployment-related configuration, including the RPS port definition and JVM parameters| + +You should not need to modify RPS configuration in a default installation. Any configuration changes you do make after you have registered the HAWQ Ranger Plug-in requires a restart of the service. You can either restart the HAWQ cluster or restart just the HAWQ Ranger Plug-in Service: + +``` shell +gpadmin@master$ /usr/local/hawq/ranger/bin/rps.sh stop +gpadmin@master$ /usr/local/hawq/ranger/bin/rps.sh start +``` + +### Changing the Frequency of Policy Caching -You may wish to change the frequency of policy caching to suit your individual needs. \ No newline at end of file +The default polling interval for HAWQ RPS policy updates is 30 seconds. To increase or decrease this value, update the `ranger.plugin.hawq.policy.pollIntervalMs` property setting in the `ranger-hawq-security.xml` file: --- End diff -- incorporated your updates, david. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #108: add config section, misc
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/108 add config section, misc added a config section. also made some minor wording edits. noticed some control-m s in the diff. not sure where those are coming from ... don't seem to affect the html generation. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ranger-addcfg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #108 commit 6881caadc01503a476810d70dabd86415a625a9f Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-30T20:46:56Z add config section, minor wording edits --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #104: HAWQ-1394 document HiveORC profile
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/104 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #106: add ranger section to logfiles page
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/106#discussion_r108961421 --- Diff: markdown/admin/logfiles.html.md.erb --- @@ -282,6 +282,12 @@ PXF provides both service- and database-level logging. Refer to [PXF Logging](.. Ambari log files may be useful in helping diagnose general cluster problems. The Ambari server log files are located in the `/var/log/ambari-server/` directory. Ambari agent log files are located in `/var/log/ambari-agent/`. Refer to [Reviewing Ambari Log Files](https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_troubleshooting/content/_reviewing_ambari_log_files.html) for additional information. +## Ranger Log Files + +The HAWQ Ranger Plug-in Service (RPS) log files may be useful in helping diagnose Ranger connectivity and authorization problems when Ranger authorization is enabled for HAWQ. You will find these log files in the `$GPHOME/ranger/plugin-service/logs/` directory. In addition to RPS service-related logs, this directory includes the `log4j` provider `audit.log` file. (Refer to [Auditing Authorization Events](../ranger/ranger-auditing.html) for information on configuring HAWQ Ranger audit logging.) --- End diff -- incorporated your requested edits, @dyozie. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #106: add ranger section to logfiles page
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/106 add ranger section to logfiles page add a section to ranger log files page with ranger and RPS log directory info. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ranger-integration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/106.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #106 commit 18205276a354915d45f62af95d9aa99178987e5a Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-30T00:21:28Z add ranger section to logfiles page --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108806756 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -30,9 +30,14 @@ The Ranger Administrative UI is installed when you install HDP. You configure th Installing or upgrading to HAWQ 2.2.0 installs the HAWQ Ranger Plug-in Service, but neither configures nor registers the plug-in. -In order to use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This is a one-time configuration that establishes connectivity to your HAWQ cluster from the Ranger Administration host. After you have registered the JAR files, you enable or disable Ranger integration in HAWQ by setting the `hawq_acl_type` configuration parameter. After Ranger integration is enabled, you must use the Ranger interface to create all security policies to manage access to HAWQ resources. Ranger is pre-populated only with several policies to allow `gpadmin` superuser access to default resources. See [Creating HAWQ Authorization Policies in Ranger](ranger-policy-creation.html) for information about creating policies in Ranger. +To use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This one-time configuration establishes connectivity to your HAWQ cluster from the Ranger Administration host. + +The `hawq_acl_type` configuration parameter allows you to shift between managing access policies through the HAWQ native interface or the Ranger policy manager. Ranger is initially started started with the `hawq_acl_type` parameter set to `standalone.` After configuring Ranger access policies, you set the `hawq_acl_type` configuration parameter to `ranger` to enable Ranger policy management. --- End diff -- as this is an intro, something like "the hawq_acl_type server configuration parameter controls the mode of authorization in place for hawq. hawq uses native authorization by default. you can enable ranger authorization with this parameter." i don't think you need to get into the values here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108807083 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -30,9 +30,14 @@ The Ranger Administrative UI is installed when you install HDP. You configure th Installing or upgrading to HAWQ 2.2.0 installs the HAWQ Ranger Plug-in Service, but neither configures nor registers the plug-in. -In order to use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This is a one-time configuration that establishes connectivity to your HAWQ cluster from the Ranger Administration host. After you have registered the JAR files, you enable or disable Ranger integration in HAWQ by setting the `hawq_acl_type` configuration parameter. After Ranger integration is enabled, you must use the Ranger interface to create all security policies to manage access to HAWQ resources. Ranger is pre-populated only with several policies to allow `gpadmin` superuser access to default resources. See [Creating HAWQ Authorization Policies in Ranger](ranger-policy-creation.html) for information about creating policies in Ranger. +To use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This one-time configuration establishes connectivity to your HAWQ cluster from the Ranger Administration host. + +The `hawq_acl_type` configuration parameter allows you to shift between managing access policies through the HAWQ native interface or the Ranger policy manager. Ranger is initially started started with the `hawq_acl_type` parameter set to `standalone.` After configuring Ranger access policies, you set the `hawq_acl_type` configuration parameter to `ranger` to enable Ranger policy management. + +Once HAWQ Ranger is enabled, access to HAWQ resources is controlled by security policies on Ranger. Access policies must be explicitly set for all groups and users, as Ranger has no knowledge of any access policies set up in the HAWQ native interface and its default is to disallow access. When first integrated, Ranger is only pre-populated with policies that allow `gpadmin` superuser access to default resources. When Ranger is enabled, you cannot manage HAWQ access through its native interface. +See [Creating HAWQ Authorization Policies in Ranger](ranger-policy-creation.html) for information about creating policies in Ranger. -The following procedures describe each configuration activity. +Perform the following procedures to configure your Ranger interface. --- End diff -- to "register the HAWQ Ranger Plug-in Service and enable Ranger authorization for HAWQ." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108809069 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -84,19 +105,28 @@ The following procedures describe each configuration activity. gpadmin@master$ hawq stop cluster --reload ``` -7. To validate connectivity between Ranger and HAWQ, access the Ranger Admin UI in Ambari, click the edit icon associated with the `hawq` service definition. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, edit your HAWQ connectivity properties directly in the Ranger Admin UI and re-test the connection. +7. When setup is complete, use the fully-qualified domain name to log into the Ambari server. Use the Ranger link in the left nav to bring up the Ranger Summary pane in the HAWQ Ambari interface. Use the Quick Links to access Ranger. This link will take you to the Ranger Login interface. + +8. Log into the Ranger Access Manager. You will see a list of icons under the Service Manager. Click the click the icon marked `hawq` under the HAWQ icon to validate connectivity between Ranger and HAWQ. A list of HAWQ policies will appear. + +9. Now return to the Service Manager and click the Edit icon on the right, under the HAWQ service icon. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, you may need to edit your Ranger connection in `pg_hba.conf,` perform + ``` bash + hawq restart cluster + ``` + and re-test the connection. ## Step 2: Configure HAWQ to Use Ranger Policy Management -The default Ranger service definition for HAWQ assigns the HAWQ user (typically `gpadmin`) all privileges to all objects. +The default Ranger service definition for HAWQ assigns the HAWQ administrator (typically `gpadmin`) all privileges to all objects. -**Warning**: If you enable HAWQ-Ranger authorization with only the default HAWQ service policies defined, other HAWQ users will have no privileges, even for HAWQ objects (databases, tables) that they own. - -1. Select the **HAWQ** Service, and then select the **Configs** tab. +Once the connection between HAWQ and Ranger is configured, you can either set up policies for the HAWQ users according to the procedures in [Creating HAWQ Authorization Policies in Ranger](ranger-policy-creation.html) or enable Ranger with only the default policies. --- End diff -- i don't think we want to imply it is ok to enable ranger with just the default policies in place. maybe we want to enhance the warning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108808162 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -84,19 +105,28 @@ The following procedures describe each configuration activity. gpadmin@master$ hawq stop cluster --reload ``` -7. To validate connectivity between Ranger and HAWQ, access the Ranger Admin UI in Ambari, click the edit icon associated with the `hawq` service definition. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, edit your HAWQ connectivity properties directly in the Ranger Admin UI and re-test the connection. +7. When setup is complete, use the fully-qualified domain name to log into the Ambari server. Use the Ranger link in the left nav to bring up the Ranger Summary pane in the HAWQ Ambari interface. Use the Quick Links to access Ranger. This link will take you to the Ranger Login interface. + +8. Log into the Ranger Access Manager. You will see a list of icons under the Service Manager. Click the click the icon marked `hawq` under the HAWQ icon to validate connectivity between Ranger and HAWQ. A list of HAWQ policies will appear. --- End diff -- not sure why we want to have them look at the policies here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108807764 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -70,9 +75,25 @@ The following procedures describe each configuration activity. gpadmin@master$ ./enable-ranger-plugin.sh -r ranger_host:6080 -u admin -p admin -h hawq_master:5432 -w gpadmin -q gpadmin ``` +***Note*** You can also enter the short form of the command: `./enable-ranger-plugin.sh -r` and the script will prompt you for entries. + When the script completes, the default HAWQ service definition is registered in the Ranger Admin UI. This service definition is named `hawq`. -6. Edit the `pg_hba.conf` file on the HAWQ master node to configure HAWQ access for \<hawq_user\> on the \<ranger-admin-node\>. For example, you would add an entry similar to the following for the example `enable-ranger-plugin.sh` call above: +6. Locate the `pg_hba.conf` file on the HAWQ master node: + +``` bash +$ hawq config --show hawq_master_directory + GUC : hawq_master_directory + Value : /data/hawq/master + $ ls /data/hawq/master --- End diff -- will listing the directory contents help the user? i find it kind of distracting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108807397 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -30,9 +30,14 @@ The Ranger Administrative UI is installed when you install HDP. You configure th Installing or upgrading to HAWQ 2.2.0 installs the HAWQ Ranger Plug-in Service, but neither configures nor registers the plug-in. -In order to use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This is a one-time configuration that establishes connectivity to your HAWQ cluster from the Ranger Administration host. After you have registered the JAR files, you enable or disable Ranger integration in HAWQ by setting the `hawq_acl_type` configuration parameter. After Ranger integration is enabled, you must use the Ranger interface to create all security policies to manage access to HAWQ resources. Ranger is pre-populated only with several policies to allow `gpadmin` superuser access to default resources. See [Creating HAWQ Authorization Policies in Ranger](ranger-policy-creation.html) for information about creating policies in Ranger. +To use Ranger for managing HAWQ authentication events, you must first install and register several HAWQ JAR files on the Ranger Administration host. This one-time configuration establishes connectivity to your HAWQ cluster from the Ranger Administration host. + +The `hawq_acl_type` configuration parameter allows you to shift between managing access policies through the HAWQ native interface or the Ranger policy manager. Ranger is initially started started with the `hawq_acl_type` parameter set to `standalone.` After configuring Ranger access policies, you set the `hawq_acl_type` configuration parameter to `ranger` to enable Ranger policy management. + +Once HAWQ Ranger is enabled, access to HAWQ resources is controlled by security policies on Ranger. Access policies must be explicitly set for all groups and users, as Ranger has no knowledge of any access policies set up in the HAWQ native interface and its default is to disallow access. When first integrated, Ranger is only pre-populated with policies that allow `gpadmin` superuser access to default resources. When Ranger is enabled, you cannot manage HAWQ access through its native interface. --- End diff -- "When Ranger authorization for HAWQ is enabled," i think the original text that was in place here looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108808070 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -84,19 +105,28 @@ The following procedures describe each configuration activity. gpadmin@master$ hawq stop cluster --reload ``` -7. To validate connectivity between Ranger and HAWQ, access the Ranger Admin UI in Ambari, click the edit icon associated with the `hawq` service definition. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, edit your HAWQ connectivity properties directly in the Ranger Admin UI and re-test the connection. +7. When setup is complete, use the fully-qualified domain name to log into the Ambari server. Use the Ranger link in the left nav to bring up the Ranger Summary pane in the HAWQ Ambari interface. Use the Quick Links to access Ranger. This link will take you to the Ranger Login interface. --- End diff -- should we just identify the direct ranger URL here? in any case, could bold the specific ambari items you are talking about. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108808330 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -84,19 +105,28 @@ The following procedures describe each configuration activity. gpadmin@master$ hawq stop cluster --reload ``` -7. To validate connectivity between Ranger and HAWQ, access the Ranger Admin UI in Ambari, click the edit icon associated with the `hawq` service definition. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, edit your HAWQ connectivity properties directly in the Ranger Admin UI and re-test the connection. +7. When setup is complete, use the fully-qualified domain name to log into the Ambari server. Use the Ranger link in the left nav to bring up the Ranger Summary pane in the HAWQ Ambari interface. Use the Quick Links to access Ranger. This link will take you to the Ranger Login interface. + +8. Log into the Ranger Access Manager. You will see a list of icons under the Service Manager. Click the click the icon marked `hawq` under the HAWQ icon to validate connectivity between Ranger and HAWQ. A list of HAWQ policies will appear. + +9. Now return to the Service Manager and click the Edit icon on the right, under the HAWQ service icon. Ensure that the Active Status is set to Enabled, and click the **Test Connection** button. You should receive a message that Ranger connected succesfully. If it fails to connect, you may need to edit your Ranger connection in `pg_hba.conf,` perform + ``` bash --- End diff -- formatting issue and spelling error (successfully) also, when updating pg_hba.conf, should be able to do a reload (don't have to restart). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #105: Reconcile Feature/ranger integration ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/105#discussion_r108807630 --- Diff: markdown/ranger/ranger-integration-config.html.md.erb --- @@ -70,9 +75,25 @@ The following procedures describe each configuration activity. gpadmin@master$ ./enable-ranger-plugin.sh -r ranger_host:6080 -u admin -p admin -h hawq_master:5432 -w gpadmin -q gpadmin ``` +***Note*** You can also enter the short form of the command: `./enable-ranger-plugin.sh -r` and the script will prompt you for entries. + When the script completes, the default HAWQ service definition is registered in the Ranger Admin UI. This service definition is named `hawq`. -6. Edit the `pg_hba.conf` file on the HAWQ master node to configure HAWQ access for \<hawq_user\> on the \<ranger-admin-node\>. For example, you would add an entry similar to the following for the example `enable-ranger-plugin.sh` call above: +6. Locate the `pg_hba.conf` file on the HAWQ master node: --- End diff -- lets use the shell prompt and formatting like previous commands. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #104: HAWQ-1394 document HiveORC profile
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/104 HAWQ-1394 document HiveORC profile include HiveORC profile info on the "Accessing Hive Data" page. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/pxforc-mainline Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit 0a4aa42ab15beebfe6b926636f72ff9dcf1b5597 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-20T19:39:57Z document HiveORC profile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #97: HAWQ-1372 - doc config change without ...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/97 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #101: HAWQ-1383 - plpgsql page cleanup, res...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/101#discussion_r105483070 --- Diff: markdown/plext/using_plpgsql.html.md.erb --- @@ -19,143 +19,283 @@ software distributed under the License is distributed on an KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---> +--> -SQL is the language of most other relational databases use as query language. It is portable and easy to learn. But every SQL statement must be executed individually by the database server. +PL/pgSQL is a trusted procedural language that is automatically installed and registered in all HAWQ databases. With PL/pgSQL, you can: -PL/pgSQL is a loadable procedural language. PL/SQL can do the following: +- Create functions +- Add control structures to the SQL language +- Perform complex computations +- Use all of the data types, functions, and operators defined in SQL -- create functions -- add control structures to the SQL language -- perform complex computations -- inherit all user-defined types, functions, and operators -- be trusted by the server +SQL is the language most relational databases use as a query language. While it is portable and easy to learn, every SQL statement is individually executed by the database server. Your client application sends each query to the database server, waits for it to be processed, receives and processes the results, does some computation, then sends further queries to the server. This back-and-forth requires interprocess communication and incurs network overhead if your client is on a different host than the HAWQ master. -You can use functions created with PL/pgSQL with any database that supports built-in functions. For example, it is possible to create complex conditional computation functions and later use them to define operators or use them in index expressions. +The PL/pgSQL language addresses some of these limitations. When creating functions with PL/pgSQL, you can group computation blocks and queries inside the database server, combining the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead. With PL/pgSQL: -Every SQL statement must be executed individually by the database server. Your client application must send each query to the database server, wait for it to be processed, receive and process the results, do some computation, then send further queries to the server. This requires interprocess communication and incurs network overhead if your client is on a different machine than the database server. +- Extra round trips between client and server are eliminated +- Intermediate, and perhaps unneeded, results do not have to be marshaled or transferred between the server and client +- Re-using prepared queries avoids multiple rounds of query parsing + -With PL/pgSQL, you can group a block of computation and a series of queries inside the database server, thus having the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead. +## PL/pgSQL Function Syntax -- Extra round trips between client and server are eliminated -- Intermediate results that the client does not need do not have to be marshaled or transferred between server and client -- Multiple rounds of query parsing can be avoided +PL/pgSQL is a block-structured language. The complete text of a function definition must be a block, which is defined as: -This can result in a considerable performance increase as compared to an application that does not use stored functions. +``` sql +[ ] +[ DECLARE +declarations ] +BEGIN +statements +END [ label ]; +``` --- End diff -- thanks for reviewing, @sansanichfb! i used the block definition identified in the postgres pl/pgsql docs. i will add a blurb about exceptions and error handling. (i didn't want to duplicate all of the postgres info on this page, my goal was to provide some introductory info and examples to get the user up and running with using pl/pgsql in common use scenarios.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #99: HAWQ-1376 - clarify pxf host and port ...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/99 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #102: a few miscellaneous updates
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/102 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #101: HAWQ-1383 - plpgsql page cleanup, res...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/101#discussion_r105472939 --- Diff: markdown/plext/using_plpgsql.html.md.erb --- @@ -19,143 +19,278 @@ software distributed under the License is distributed on an KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ---> +--> -SQL is the language of most other relational databases use as query language. It is portable and easy to learn. But every SQL statement must be executed individually by the database server. +PL/pgSQL is a trusted procedural language that is automatically installed and registered in all HAWQ databases. With PL/pgSQL, you can: -PL/pgSQL is a loadable procedural language. PL/SQL can do the following: +- Create functions +- Add control structures to the SQL language +- Perform complex computations +- Use all of the data types, functions, and operators defined in SQL -- create functions -- add control structures to the SQL language -- perform complex computations -- inherit all user-defined types, functions, and operators -- be trusted by the server +SQL is the language most relational databases use as a query language. While it is portable and easy to learn, every SQL statement is individually executed by the database server. Your client application sends each query to the database server, waits for it to be processed, receives and processes the results, does some computation, then sends further queries to the server. This back-and-forth requires interprocess communication and incurs network overhead if your client is on a different host than the HAWQ master. -You can use functions created with PL/pgSQL with any database that supports built-in functions. For example, it is possible to create complex conditional computation functions and later use them to define operators or use them in index expressions. +PL/pgSQL does not have these limitations. When creating functions with the PL/pgSQL language, you can group computation blocks and queries inside the database server, combining the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead. With PL/pgSQL: -Every SQL statement must be executed individually by the database server. Your client application must send each query to the database server, wait for it to be processed, receive and process the results, do some computation, then send further queries to the server. This requires interprocess communication and incurs network overhead if your client is on a different machine than the database server. +- Extra round trips between client and server are eliminated +- Intermediate, and perhaps unneeded, results do not have to be marshaled or transferred between the server and client +- You avoid multiple rounds of query parsing + -With PL/pgSQL, you can group a block of computation and a series of queries inside the database server, thus having the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead. +## PL/pgSQL Function Syntax -- Extra round trips between client and server are eliminated -- Intermediate results that the client does not need do not have to be marshaled or transferred between server and client -- Multiple rounds of query parsing can be avoided +PL/pgSQL is a block-structured language. The complete text of a function definition must be a block, which is defined as: -This can result in a considerable performance increase as compared to an application that does not use stored functions. +``` sql +[ ] +[ DECLARE +declarations ] +BEGIN +statements +END [ label ]; +``` -PL/pgSQL supports all the data types, operators, and functions of SQL. +Each declaration and each statement within a block is terminated by a semicolon. A block that appears within another block must have a semicolon after `END`, as shown above; however the final `END` that concludes a function body does not require a semicolon. + +You can specify all key words and identifiers in mixed upper and lower case. Identifiers are implicitly converted to lowercase unless double-quoted. + +PL/pgSQL supports two types of comments. A double dash (`--`) starts a comment that extends to the end of the line. A `/*` starts a block comment that extends to the next occurrence of `*/`. Block comments cannot be nested, but you can enclose double dash comments into a block comment and a double dash can hide the block comment delimiters `/*` and `*/`. + +This example PL/pgSQL function adds th
[GitHub] incubator-hawq-docs pull request #97: HAWQ-1372 - doc config change without ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/97#discussion_r105307485 --- Diff: markdown/admin/ambari-admin.html.md.erb --- @@ -206,44 +208,22 @@ First ensure that the new node(s) has been configured per the instructions found 23. (Optional.) If you enabled temporary password-based authentication while preparing/configuring your HAWQ host systems, turn off password-based authentication as described in [Apache HAWQ System Requirements](../requirements/system-requirements.html#topic_pwdleh). - Manually Updating the HAWQ Configuration -If you need to expand your HAWQ cluster without restarting the HAWQ service, follow these steps to manually apply the new HAWQ configuration. (Use these steps *instead* of following Step 7 in the above procedure.): + Cluster Expansion Workaround +If you need to expand your HAWQ cluster without restarting the HAWQ service, follow this procedure to manually apply the HAWQ configuration changes required for this operation. (Perform these steps *instead of* following Step 9b in the above procedure.) -1. Update your configuration to use the new `default_hash_table_bucket_number` value that you calculated: - 1. SSH into the HAWQ master host as the `gpadmin` user: -```shell -$ ssh gpadmin@ -``` - 2. Source the `greenplum_path.sh` file to update the shell environment: -```shell -$ source /usr/local/hawq/greenplum_path.sh -``` - 3. Verify the current value of `default_hash_table_bucket_number`: -```shell -$ hawq config -s default_hash_table_bucket_number -``` - 4. Update `default_hash_table_bucket_number` to the new value that you calculated: -```shell -$ hawq config -c default_hash_table_bucket_number -v -``` - 5. Reload the configuration without restarting the cluster: -```shell -$ hawq stop cluster -u -``` - 6. Verify that the `default_hash_table_bucket_number` value was updated: -```shell -$ hawq config -s default_hash_table_bucket_number -``` -2. Edit the `/usr/local/hawq/etc/slaves` file and add the new HAWQ hostname(s) to the end of the file. Separate multiple hosts with new lines. For example, after adding host4 and host5 to a cluster already contains hosts 1-3, the updated file contents would be: +1. Perform the procedure identified in [Updating HAWQ Configuration Without Cluster Restart](#manual-cfg-norestart) to update your configuration to use the new `default_hash_table_bucket_number` value that you calculated and set via the Ambari UI. You may skip Step 1 of this procedure. --- End diff -- yes, repeating the config steps in the workaround is much clearer for the user. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #97: HAWQ-1372 - doc config change without ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/97#discussion_r105305989 --- Diff: markdown/admin/ambari-admin.html.md.erb --- @@ -368,6 +348,49 @@ All of the listed steps are mandatory. This ensures that HAWQ service remains fu 4. Restart HAWQ service to propagate the configuration change to all Ambari agents. This will synchronize the password on the host machines with the password that you specified in Ambari. + +## Updating HAWQ Configuration Without Cluster Restart +There may be circumstances, such as during dynamic cluster expansion, when you must update HAWQ configuration parameters but cannot tolerate cluster downtime in your Ambari-managed cluster. In these situations, perform the following workaround to update and reload HAWQ configuration changes without a complete cluster restart. + +**Note**: Use this procedure for setting or updating only those HAWQ server configuration parameters with a set classification of reload. + +1. Use the Ambari console UI to set and save the desired HAWQ configuration parameter(s) via the **HAWQ** service **Settings**, **Configs >Settings**, and/or **Configs >Advanced** tabs. Make sure to note the configuration parameter names and new values. You may need to hover the mouse over the configuration field to identify the HAWQ parameter name. + +**Note**: After updating the configuration parameters, do *not* restart the HAWQ service. + +1. Use HAWQ command line utilities to update the same HAWQ configuration parameters: +1. Log in to the HAWQ master host as a HAWQ administrator and source greenplum_path.sh: + +``` shell +$ ssh gpadmin@ +gpadmin@master$ source /usr/local/hawq/greenplum_path.sh +``` + +2. Identify the current value of the HAWQ configuration parameter: + +``` shell +gpadmin@master$ hawq config -s +``` + +3. Set the configuration parameter to the new value: + +``` shell +gpadmin@master$ hawq config -c -v +``` + +Perform Steps 2 and 3 for each configuration parameter you set or updated via Ambari. + +3. Reload the HAWQ configuration; this operation does not restart the cluster: + +``` shell +gpadmin@master$ hawq stop cluster --reload +``` + +4. Verify that each parameter was updated: + +```shell +gpadmin@master$ hawq config -s +``` --- End diff -- can you clarify what you are looking for here, @dyozie? the config was changed first in ambari (local only) and then via the CLI (changed in running cluster). when ambari is restarted, it should effectively then know about the change that was made via the CLI. i don't think it would be new information, would it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #102: a few miscellaneous updates
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/102 a few miscellaneous updates did not create a jira for these minor updates: - move "SQL Commands" subnav to the first entry under "HAWQ Reference" - update kerberos principal in an example You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/lo-misc-updates-11 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #102 commit 0c89035e29aa0f761f23643bded9b9c9d82db05f Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-09T23:57:09Z move SQL Commands subnav to first under HAWQ Reference commit 2ff8a8e4e341f89a59d714762311a7671b6ba31a Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-09T23:58:40Z change kerberos principal to HDB.LOCAL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #99: HAWQ-1376 - clarify pxf host and port ...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/99#discussion_r105300059 --- Diff: markdown/admin/HAWQFilespacesandHighAvailabilityEnabledHDFS.html.md.erb --- @@ -240,3 +240,7 @@ For command-line administrators: $ hawq init standby -n -M fast ``` + +## Using PXF with HDFS NameNode HA + +When HDFS NameNode High Availability is enabled, identify the HDFS Nameservice ID in the `LOCATION` clause \<host\> field when invoking any PXF `CREATE EXTERNAL TABLE` command. If the \<port\> is omitted from the `LOCATION` URI, PXF connects to the port number designated by the `pxf_service_port` server configuration parameter value (default is 51200). --- End diff -- thanks for reviewing @dyozie. i've pushed your requested change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #101: HAWQ-1383 - plpgsql page cleanup, res...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/101 HAWQ-1383 - plpgsql page cleanup, restructure, more examples many changes to the plpgsql page: - restructured and added some general info - added links to postgres docs where appropriate - added several concepts sections with examples - moved polymorphic types section to data types page. also pared down the info, hawq doesn't appear to support anynonarray and anyenum - added polymorphic example - removed variadic function argument section - tried several incantations and couldn't get it to work. either hawq doesn't support variadic functions (hard to tell from the code) or user error. if the latter i can add a section/example back in. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1383-plpgsql-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #101 commit d48c238a690a3e102f789a01e9ea3e6e62861310 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-09T02:49:31Z HAWQ-1383 - plpgsql page cleanup, restructure, more examples --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #99: HAWQ-1376 - clarify pxf host and port ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/99 HAWQ-1376 - clarify pxf host and port description clarify the pxf host and port description (create external table, location clause URI) across the docs - host - any pxf node - port - the pxf port. if omitted, HDFS high availability is assumed and host is taken to be the HDFS nameservice id, and pxf_service_port value is used for the port (default 51200) also added "using pxf with HA enabled HDFS" section to HA doc, identifying HA implications on PXF. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1376-pxfclarify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/99.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #99 commit 12d0f4be1b3181c46bf322a7be5c0e9767cdd843 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-07T01:12:15Z HAWQ-1376 - clarify pxf host and port description --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #98: HAWQ-1296 - initial draft of hawq gett...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/98 HAWQ-1296 - initial draft of hawq getting started guide initial revision of the "Getting Started with HAWQ" tutorial. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/getting-started2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/98.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #98 commit fb21d13cd8970bf7ab816c1e51ef6ec0702cd035 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-25T15:28:00Z initial check-in of hawq getting started guide --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #97: HAWQ-1372 - doc config change without ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/97 HAWQ-1372 - doc config change without cluster restart for ambari-managed clusters cluster restart for config change in an ambari-managed deployment may not be tolerated in some cases. document the procedure to update hawq config and reload for ambari-managed clusters. also including a small rework to master mirroring standby resynchronization section with this PR. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1372-ambari-no-restart Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/97.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #97 commit 1ac4e5bae744e6771f66ec0157914cf7fd593485 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-02T22:12:52Z HAWQ-1372 - ambari cfg change w/o cluster restart commit 9610a267406f9c06722aeced484152e6ce76a877 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-03-02T22:51:02Z rework master mirror standby resync section --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #90: HAWQ-1289 - add hive profile example w...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/90 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #94: HAWQ-1304 - multiple doc changes for P...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/94 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #94: HAWQ-1304 - multiple doc changes for P...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/94#discussion_r100115473 --- Diff: markdown/pxf/ReadWritePXF.html.md.erb --- @@ -131,6 +149,8 @@ Note: The DELIMITER parameter is mandatory. +**Notes**: Metadata identifies the Java class that provides field definitions in the relation. OutputFormat identifies the file format for which a specific profile is optimized. While the built-in `Hive*` profiles provide Metadata and OutputFormat classes, most profiles will have no need to implement or specify these classes. --- End diff -- the statement is kind of vague, yes. i will make it clearer. it seems to me that the outputFormat property is more admin- or developer-focused. what i mean here is that a typical end-user using a Hive* profile probably does not need to know about the setting. an admin configuring custom profiles or a developer creating a custom plug-in may be interested. the current admin- and developer- focused PXF docs need some attention. can we address a more detailed discussion of this property as part of that rework at a later time? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #94: HAWQ-1304 - multiple doc changes for P...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/94 HAWQ-1304 - multiple doc changes for PXF and Hive Plugin doc changes for HAWQ-1228: - outputformat class - hcatalog use of optimal Hive* profile - hive profile use of optimal Hive* profile - enabling logging to see Hive* profile actually used - remove performance statements related changes: - metadata class - clarify Hive plug-in prerequisites - add example PXF profile definition - misc editting and formatting changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1304-pxf-n-hive-chgs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/94.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #94 commit d0fb9ee40dbfa1f2dfc231f491e0cfd20b5dba8a Author: Lisa Owen <lo...@pivotal.io> Date: 2017-02-02T23:22:28Z multiple doc changes for PXF and Hive Plugin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #90: HAWQ-1289 - add hive profile example w...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/90#discussion_r97899104 --- Diff: markdown/pxf/HivePXF.html.md.erb --- @@ -172,7 +188,7 @@ Hive-plug-in-specific keywords and values used in the [CREATE EXTERNAL TABLE](.. ## Hive Profile -The `Hive` profile works with any Hive file format. It can access heterogenous format data in a single table where each partition may be stored as a different file format. +The `Hive` profile works with any Hive file format. This profile can also be used to access heterogenous format data in a single table where each partition may be stored in a different file format. Refer to the [Partition Filtering](#partitionfiltering) discussion later in this topic for additional information on partitioning and the `Hive` profile. --- End diff -- how about "You can use this profile to access heterogenous format data in a single table where each partition may be stored in a different file format." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #91: HAWQ-1293 - identify/tag deprecated gu...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/91 HAWQ-1293 - identify/tag deprecated gucs identifying the documented deprecated gucs as such. these include work_mem and gp_interconnect_setup_timeout. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/deprecated-gucs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/91.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #91 commit 691f4ce503ff692ae3258163ee59536469bab056 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-25T18:39:14Z HAWQ-1293 - identify/tag deprecated gucs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #88: HAWQ-1261 - add discussion of HAWQ adm...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/88 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #88: HAWQ-1261 - add discussion of HAWQ adm...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/88#discussion_r97174865 --- Diff: markdown/admin/logfiles.html.md.erb --- @@ -0,0 +1,272 @@ +--- +title: HAWQ Administrative Log Files +--- + +Log files are files that include messages and other information about your HAWQ deployment, including the database and utilities. + +Every database instance in HAWQ \(master, standby, and segments\) runs a PostgreSQL database server with its own server log file. You generate log files when you invoke HAWQ management utilities directly, or indirectly via Ambari management operations. Additionally, other components in your HAWQ cluster (PXF, HDFS) generate log files of their own. + +These log files are distinctly located, formatted, configured, and managed. + +**Locating Log Files** + +HAWQ administrative log files reside in pre-defined or configured locations on the local file system of the HAWQ node. + +**Configuring Logging Parameters** + +Configurable logging parameters may affect what, when, and where messages are logged. You configure HAWQ administrative logging options via HAWQ server configuration parameters or command line options. + +**Managing Log Files** + +Log files are created and/or rotated at pre-defined or configured intervals. Note that administrative log files are not automatically truncated or deleted. The administrator must implement and periodically run scripts to clean up these log files. + +## HAWQ Database Server Log Files + + +### Locating HAWQ Log Files + +Each HAWQ master, standby, and segment database instance has its own server log file. Daily log files are created in the `pg_log` subdirectory of the master and segment data directory on the respective HAWQ node. You can obtain the master data directory location from the `hawq_master_directory` property value set in the `$GPHOME/etc/`[`hawq-site.xml`](../reference/HAWQSampleSiteConfig.html) configuration file. Similarly, you can obtain the segment data directory location from the `hawq_segment_directory` property value from `hawq-site.xml`. + +The naming convention for HAWQ database server log files is `hawq-_.[csv|log]`. For example, `hawq-2017-01-02_061611.csv` or `hawq-2017-01-03_001704.log`. The number and size of log files present for a given \<date\> is dependent upon the values of certain HAWQ server configuration parameters (discussed later in this document). + +### HAWQ Log Format + +The HAWQ server log files are written in text or comma-separated values \(CSV\) format. + +HAWQ log entries may include the following fields: + + + #Field NameData TypeDescription + 1event_timetimestamp with time zoneTh time that the log entry was written to the log + 2user_namevarchar(100)The database user name + 3database_namevarchar(100)The database name + 4process_idvarchar(10)The system process ID (prefixed with "p") + 5thread_idvarchar(50)The thread count (prefixed with "th-") + 6remote_hostvarchar(100)The hostname/address of the client machine (if on master node). The hostname/address of the master (if on segment node). + 7remote_portvarchar(10)The segment or master port number + 8session_start_timetimestamp with time zoneThe time the session connection was opened + 9transaction_idintThe top-level transaction ID on the master; this ID is the parent of any subtransactions. + 10gp_session_idtextThe session identifier number (prefixed with "con") + 11gp_command_counttextThe command number within a session (prefixed with "cmd") + 12gp_segmenttextThe segment content identifier. The master always has a content ID of -1. + 13slice_idtextThe slice ID (portion of the query plan being executed) + 14distr_tranx_idtextThe distributed transaction identifier + 15local_tranx_idtextThe local transaction identifier + 16sub_tranx_idtextThe subtransaction identifier + 17event_severityvarchar(10)The event severity; values include: LOG, ERROR, FATAL, PANIC, DEBUG1, DEBUG2 + 18sql_state_codevarchar(10)The SQL state code associated with the log message + 19event_messagetextThe log or error message text + 20event_detailtextThe detail message text associated with an error or warning message + 21event_hinttextThe hint message text associated with an error or warning message + 22internal_querytextThe internally-generated query text + 23internal_query_posintThe cursor index into the internally-generated query text + 24event_contexttextThe context in which this message is generated + 25debug_query_stringtextUser-supplied query string with full detail for debugging. This string can be modified for internal use
[GitHub] incubator-hawq-docs pull request #90: HAWQ-1289 - add hive profile example w...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/90 HAWQ-1289 - add hive profile example with partitioning and mixed data types - added the requested example - restructured the "Partition Filtering" section a bit to better incorporate the new example You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1289-hive-part-multiform Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/90.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #90 commit 8c503d4bd752eb84e26a01c6e204758f15139167 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-20T22:42:27Z hive profile - add example for partioning w mixed data types --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #46: HAWQ-1119 - create doc content for PXF...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/46 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #46: HAWQ-1119 - create doc content for PXF...
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/46#discussion_r96705982 --- Diff: markdown/pxf/HDFSWritablePXF.html.md.erb --- @@ -396,7 +394,7 @@ The field type of `recordkey` must correspond to the key type, much as the other ### Example: Using Record Keys -Create an external readable table to access the record keys from the writable table `pxf_tbl_seqwrit` that you created in [Example: Writing Data Using the SequenceWritable Profile](#profile_hdfsseqwritable_writing). The `recordkey` is of type `int8`. (We did not identify a record key when creating the writable table. +Create an external readable table to access the record keys from the writable table `pxf_tbl_seqwrit` that you created in [Example: Writing Data Using the SequenceWritable Profile](#profile_hdfsseqwritable_writing). The `recordkey` is of type `int8`. (You did not identify a record key when you created the writable table. --- End diff -- those two sentences are not directly related. the type of recordkey will always be int8. i am going to just remove the "You did not identify a record key..." sentence as a variant is used after the example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #86: HAWQ-1263 - misc doc updates
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/86 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #86: HAWQ-1263 - misc doc updates
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/86#discussion_r96321794 --- Diff: markdown/ddl/ddl-database.html.md.erb --- @@ -2,77 +2,75 @@ title: Creating and Managing Databases --- -A HAWQ system is a single instance of HAWQ. There can be several separate HAWQ systems installed, but usually just one is selected by environment variable settings. See your HAWQ administrator for details. - -There can be multiple databases in a HAWQ system. This is different from some database management systems \(such as Oracle\) where the database instance *is* the database. Although you can create many databases in a HAWQ system, client programs can connect to and access only one database at a time â you cannot cross-query between databases. +Your HAWQ deployment may have multiple databases. This is different from some database management systems \(such as Oracle\) where the database instance *is* the database. Although you can create many databases in a HAWQ system, client programs can connect to and access only one database at a time â you cannot cross-query between databases. ## About Template Databases -Each new database you create is based on a *template*. HAWQ provides a default database, *template1*. Use *template1* to connect to HAWQ for the first time. HAWQ uses *template1* to create databases unless you specify another template. Do not create any objects in *template1* unless you want those objects to be in every database you create. +Each new database you create is based on a *template*. HAWQ provides a default database, `template1`. HAWQ uses `template1` to create databases unless you specify another template. Do not create any objects in `template1` unless you want those objects to be in every database you create. -HAWQ uses two other database templates, *template0* and *postgres*, internally. Do not drop or modify *template0* or *postgres*. You can use *template0* to create a completely clean database containing only the standard objects predefined by HAWQ at initialization, especially if you modified *template1*. +HAWQ uses two other database templates internally, `template0` and `postgres`. Do not drop or modify `template0` or `postgres`. You can use `template0` to create a completely clean database containing only the standard objects predefined by HAWQ at initialization, especially if you have modified `template1`. --- End diff -- i think template0 and template1 start out the same. you cannot modify template0, but you can template1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #86: HAWQ-1263 - misc doc updates
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/86 HAWQ-1263 - misc doc updates this PR includes: - AWS configuration document updates including clarifying storage options and security groups - minor gpfdist example update for single instance You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/lo-misc-7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/86.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #86 commit dbca6f3434d28c075051887d93803a74d202b7b0 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-13T02:52:01Z aws config updates commit 6113db3fcb270caef0f5382cb883491b38e7e22e Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-13T20:35:11Z fix gpfdist single instance example --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #83: HAWQ-1252 - pljava doc cleanup, workin...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/83 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #84: HAWQ-1263 - clarifications to ambari (...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/84 HAWQ-1263 - clarifications to ambari (inc subnav), hawq op env misc updates to ambari docs and intro to hawq env page. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/lo-misc-6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/84.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #84 commit d18098281c986b43639d22ecb5e75a63b1f4bfc8 Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-09T23:30:55Z HAWQ-1263 - clarifications to ambari (inc subnav), hawq op env --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #83: HAWQ-1252 - pljava doc cleanup, workin...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/83 HAWQ-1252 - pljava doc cleanup, working examples MANY changes to this file. restructured content a bit. all of the examples are now runnable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1252-pljava-review Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/83.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #83 commit 56331647e17ac6556ce98d1d9500ddd07ab3986e Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-05T00:31:37Z pljava doc cleanup, working examples --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #80: HAWQ-1209 - hawq overview page
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/80 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #79: HAWQ-1163 - add subnav for ambari rest...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/79 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #78: HAWQ-1163 - add new topic on ambari re...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/78 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #77: HAWQ-1216 - clean up plpython docs
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/77 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #70: HAWQ-1202 - PXF server config params
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/70 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #71: HAWQ-1202 - subnav changes for new pxf...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/71 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #82: HAWQ-1253 - monitoring section add xre...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/82 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #82: HAWQ-1253 - monitoring section add xre...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/82 HAWQ-1253 - monitoring section add xref to hawq_toolkit reviewed the page: - moved hawq_toolkit section from near end to near front - it was referenced multiple times before its discussion - added a link to hawq_toolkit reference section - reviewed and updated examples to make sure they work - multiple wording changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1253-monitor-hawq_toolkit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/82.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #82 commit d96e8d2b49170903d13f94c95f02eea02d3baefb Author: Lisa Owen <lo...@pivotal.io> Date: 2017-01-04T17:41:31Z review/update page, add xref to hawq_toolkit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #77: HAWQ-1216 - clean up plpython docs
Github user lisakowen commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94606714 --- Diff: plext/using_plpython.html.md.erb --- @@ -2,374 +2,608 @@ title: Using PL/Python in HAWQ --- -This section contains an overview of the HAWQ PL/Python language extension. +This section provides an overview of the HAWQ PL/Python procedural language extension. ## About HAWQ PL/Python -PL/Python is a loadable procedural language. With the HAWQ PL/Python extension, you can write HAWQ user-defined functions in Python that take advantage of Python features and modules to quickly build robust database applications. +PL/Python is embedded in your HAWQ product distribution or within your HAWQ build if you chose to enable it as a build option. + +With the HAWQ PL/Python extension, you can write user-defined functions in Python that take advantage of Python features and modules, enabling you to quickly build robust HAWQ database applications. HAWQ uses the system Python installation. ### HAWQ PL/Python Limitations -- HAWQ does not support PL/Python triggers. +- HAWQ does not support PL/Python trigger functions. - PL/Python is available only as a HAWQ untrusted language. ## Enabling and Removing PL/Python Support -To use PL/Python in HAWQ, you must either use a pre-compiled version of HAWQ that includes PL/Python or specify PL/Python as a build option when compiling HAWQ. +To use PL/Python in HAWQ, you must either install a binary version of HAWQ that includes PL/Python or specify PL/Python as a build option when compiling HAWQ from source. + +PL/Python user-defined functions (UDFs) are registered at the database level. To create and run a PL/Python UDF on a database, you must register the PL/Python language with the database. + +On every database to which you want to install and enable PL/Python: + +1. Connect to the database using the `psql` client: + +``` shell +$ psql -d +``` + +Replace \<dbname\> with the name of the target database. + +2. Run the following SQL command to register the PL/Python procedural language; you must be a database superuser to register new languages: + +``` sql +dbname=# CREATE LANGUAGE plpythonu; +``` -To create and run a PL/Python user-defined function (UDF) in a database, you must register the PL/Python language with the database. On every database where you want to install and enable PL/Python, connect to the database using the `psql` client. +**Note**: `plpythonu` is installed as an *untrusted* language; it offers no way of restricting what you can program in UDFs created with the language. --- End diff -- yes, it is kind of confusing. the operations one can perform with trusted languages are restricted; unprivileged database users can be permitted to create and run UDFs created in these languages. untrusted languages are the opposite - you can do pretty much anything, and i think are restricted to superusers. (will verify.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #81: HAWQ-1209 - add subnav for setting up ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/81 HAWQ-1209 - add subnav for setting up hawq op env created a new page called "Introducing the HAWQ Operating Environment". this new page is located after "Overview" in the "Running a HAWQ Cluster" subnav. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/subnav-hawq-openv-intro Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/81.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #81 commit b30e4cc7670c76d9d53c159a940f2facb48155b0 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-30T16:08:50Z add subnav for setting up hawq op env --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #79: HAWQ-1163 - add subnav for ambari rest...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/79 HAWQ-1163 - add subnav for ambari rest api topic include new ambari rest api topic in subnav You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/subnav-ambari-rest-api Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/79.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #79 commit 0477e213b0a1510060c202847064dcb2a11d10bb Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-29T22:45:41Z add subnav for ambari rest api topic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #78: HAWQ-1163 - add new topic on ambari re...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/78 HAWQ-1163 - add new topic on ambari rest api an introduction to using the ambari rest api for hawq/pxf management activities. not meant to provide comprehensive topic coverage. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/ambari-rest-api Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/78.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #78 commit c58ed12d4a9ecf15030c1b3b16f8d138bad4248c Author: Lisa Owen <lo...@pivotal.io> Date: 2016-11-18T01:01:31Z add file for new topic on ambari rest api --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #77: HAWQ-1216 - clean up plpython docs
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/77 HAWQ-1216 - clean up plpython docs restructured content, updated examples, added new examples You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1216-plpython-review Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/77.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #77 commit 881ad95ab12ab745894e54fe40fe9c935f6f6237 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-13T22:19:31Z clean up plpython docs - working examples, reorg --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #76: HAWQ-1218 - add subnav entry for DROP ...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/76 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #75: HAWQ-1218 - add drop language sql refe...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/75 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #74: HAWQ-1219 - add ao and parquet relstor...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/74 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #76: HAWQ-1218 - add subnav entry for DROP ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/76 HAWQ-1218 - add subnav entry for DROP LANGUAGE add subnav entry for new DROP LANGUAGE ref page You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/subnav-HAWQ-1218-drop-language Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/76.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #76 commit e2a273ea306a0320319ec50714c2d73a2671eb67 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-14T18:41:12Z add subnav entry for DROP LANGUAGE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #75: HAWQ-1218 - add drop database sql refe...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/75 HAWQ-1218 - add drop database sql reference page content for DROP LANGUAGE SQL reference page. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1218-drop-language Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/75.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #75 commit 897333272fee41b7fee3ae75c269ace957e941c8 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-14T18:38:04Z add drop database sql reference page --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #73: HAWQ-1192 follow-up - simplify config ...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/73 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #74: HAWQ-1219 - add ao and parquet relstor...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/74 HAWQ-1219 - add ao and parquet relstorage types add parquet and append only restorage types to pg_class reference page (table) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/misc-updates-lo5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/74.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #74 commit 6bd32a39035a58cca66b5dbac0e165f126d6ca47 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-12-09T22:26:24Z add ao and parquet relstorage types --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #73: HAWQ-1192 follow-up - simplify config ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/73 HAWQ-1192 follow-up - simplify config change steps simplify the config change steps to case where PXF initialized and started. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/HAWQ-1192-more Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/73.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #73 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #69: HAWQ-1192 - add discussion of heap and...
Github user lisakowen closed the pull request at: https://github.com/apache/incubator-hawq-docs/pull/69 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq-docs pull request #71: HAWQ-1202 - add new pxf server config ...
GitHub user lisakowen opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/71 HAWQ-1202 - add new pxf server config param subcategory and missing pxf params subnav updates for PXF server config params. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lisakowen/incubator-hawq-docs feature/subnav-pxf-gucs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit e09f676b1c15e9a48d8cf6040703ef7d88043452 Author: Lisa Owen <lo...@pivotal.io> Date: 2016-10-31T19:22:32Z add new pxf guc subcategory and add pxf gucs to subnav --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
