[GitHub] incubator-hawq issue #1071: HAWQ-1242. hawq-site.xml default content has wro...

[radarwave]

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/1071
  
LGTM.

Just wondering how hawq read these mis-match GUCs before you change them? 
Seems they still can get the right values.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...

[stanlyxiang]

Github user stanlyxiang commented on the issue:

https://github.com/apache/incubator-hawq/pull/1073
  
cc @radarwave  . For Radar used to build docker image for concourse 
pipeline, maybe he has some comments. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

Github user guofengrichard commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525426
  
--- Diff: contrib/hawq-docker/README.md ---
@@ -0,0 +1,96 @@
+# hawq-docker
+
+hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker 
images and scripts to help developers of Apache HAWQ to setup building and 
testing environment with docker.
+
+Both CentOS 7 and CentOS 6 are supported.
+Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to 
switch between CentOS 7 and CentOS 6.
+
+Take CentOS 7 as an example below.
+
+# Install docker
+* following the instructions to install docker.
+https://docs.docker.com/
+
+# Setup build and test environment
+* clone this repository
+```
+git clone https://github.com/guofengrichard/hawq-docker.git .
+```
+* Get the docker images
+```
+  cd hawq-docker
+  make pull (recommended)
+OR
+  make build
+``` 
+* setup a 5 nodes virtual cluster for Apache HAWQ build and test.
+```
+make run
+```
+Now let's have a look about what we creted.
+```
+[root@localhost hawq-docker]# docker ps -a
+CONTAINER IDIMAGE  COMMAND
CREATED STATUS  PORTS   NAMES
+382b2b3360d1hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode3
+86513c331d45hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode2
+c0ab10e46e4ahawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode1
+e27beea63953hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-namenode
+1f986959bd04hawq/hawq-dev:centos7"/bin/true"2 
minutes ago   Created centos7-data
+```
+**centos7-data** is a data container and mounted to /data directory on all 
other containers to provide a shared storage for the cluster. 
+
+# Build and Test Apache HAWQ
+* attach to namenode
+```
+docker exec -it centos7-namenode bash
+```
+* check if HDFS working well
+```
+sudo -u hdfs hdfs dfsadmin -report
+```
+* clone Apache HAWQ code to /data direcotry
+```
+git clone https://github.com/apache/incubator-hawq.git /data/hawq
+```
+* build Apache HAWQ
+```
+cd /data/hawq
+./configure --prefix=/data/hawq-dev
+make
+make install
+```
+(When you are using CentOS 6, run command `scl enable devtoolset-2 bash` 
before
+configuring hawq and run command `exit` after installing hawq.) 
--- End diff --

The 'exit' command is to exit from the bash environment setup by previous 
'scl' command.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

Github user guofengrichard commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525211
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with 
other containers
+LOCAL := 
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all: 
+   @echo " Usage:"
+   @echo "To setup a build and test environment: make run"
+   @echo "To start all containers:   make start"
+   @echo "To stop all containers:make stop"
+   @echo "To remove hdfs containers: make clean"
+   @echo "To remove all containers:  make 
distclean"
+   @echo ""
+   @echo "To build images locally:   make build"
+   @echo "To pull latest images: make pull"
+
+build:
+   @make -f $(THIS_MAKEFILE_PATH) build-hawq-dev-$(OS_VERSION)
+   @make -f $(THIS_MAKEFILE_PATH) build-hawq-test-$(OS_VERSION)
+   @echo "Done!"
--- End diff --

That's right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

Github user guofengrichard commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525000
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with 
other containers
+LOCAL := 
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all: 
+   @echo " Usage:"
+   @echo "To setup a build and test environment: make run"
+   @echo "To start all containers:   make start"
+   @echo "To stop all containers:make stop"
+   @echo "To remove hdfs containers: make clean"
+   @echo "To remove all containers:  make 
distclean"
+   @echo ""
+   @echo "To build images locally:   make build"
--- End diff --

No cleanup command for 'make build'. Removing docker images is not 
encouraged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

Github user guofengrichard commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94524665
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with 
other containers
+LOCAL := 
--- End diff --

Yes. '/data' is in containers and will be shared among all the containers.
'LOCAL' is supposed to be a local directory on the host. If it is defined, 
the directory will be mounted to '/data' in all containers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

Github user guofengrichard commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94524175
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
--- End diff --

@stanlyxiang is right about NDATANODES and CUR_DATANODE.

Good suggestion on configuring the number. For now user can define the 
value via parameters to 'make', for example
 $ make NDATANODES=6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1245) can HAWQ support alternate python module deployment directory?

[Paul Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo updated HAWQ-1245:
---
Component/s: (was: Core)
 Command Line Tools

> can HAWQ support alternate python module deployment directory?
> --
>
> Key: HAWQ-1245
> URL: https://issues.apache.org/jira/browse/HAWQ-1245
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Lisa Owen
>Assignee: Ed Espino
>Priority: Minor
>
> HAWQ no longer embeds python and is now using the system python installation. 
>  with this change, installing a new python module now requires root/sudo 
> access to the system python directories.  is there any reason why HAWQ would 
> not be able to support deploying python modules to an alternate directory 
> that is owned by gpadmin?  or using a python virtual environment?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[stanlyxiang]

Github user stanlyxiang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94522816
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
--- End diff --

I think NDATANODES is the total number of datanodes. And CUR_DATANODE is a 
iteration placeholder during the loop for start/stop/run/other operations for 
the hdfs.  

I think a configurable number of datanodes are more reasonable. And you can 
set 3 as default value. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...

[paul-guo-]

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1073
  
By the way, what is the difference between hawq-dev and hawq-test? (One for 
dev environment and another includes hdfs and/or hawq? It seems that there is 
no comment for this), And it seems many configuration files comes from hdfs. 
Why not use the default configuration files and configure them later in CLI?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1071: HAWQ-1242. hawq-site.xml default content has wro...

[paul-guo-]

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1071
  
1) Please make sure the doc has corresponding change if needed (notify them 
if needed). If I remember correctly, dfs_url is still in that doc.
2) Please make the commit message begin with HAWQ-1242 since we have a 
script, which needs this pattern, to detect the Apache JIRA status.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1071: HAWQ-1242. hawq-site.xml default content has wro...

[paul-guo-]

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/1071
  
Otherwise +1. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796695#comment-15796695
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94512401
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
+PL/Python user-defined functions (UDFs) are registered at the database 
level. To create and run a PL/Python UDF on a database, you must register the 
PL/Python language with the database. 
+
+On every database to which you want to install and enable PL/Python:
+
+1. Connect to the database using the `psql` client:
+
+``` shell
+$ psql -d 
+```
+
+Replace \ with the name of the target database.
+
+2. Run the following SQL command to register the PL/Python procedural 
language; you must be a database superuser to register new languages:
+
+``` sql
+dbname=# CREATE LANGUAGE plpythonu;
+```
 
-To create and run a PL/Python user-defined function (UDF) in a database, 
you must register the PL/Python language with the database. On every database 
where you want to install and enable PL/Python, connect to the database using 
the `psql` client.
+**Note**: `plpythonu` is installed as an *untrusted* language; it 
offers no way of restricting what you can program in UDFs created with the 
language.
 
-```shell
-$ psql -d 
+To remove support for `plpythonu` from a database, run the following SQL 
command; you must be a database superuser to remove a registered procedural 
language:
+
+``` sql
+dbname=# DROP LANGUAGE plpythonu;
 ```
 
-Replace \ with the name of the target database.
+## Developing Functions with PL/Python 
+
+PL/Python functions are defined using the standard SQL [CREATE 
FUNCTION](../reference/sql/CREATE-FUNCTION.html) syntax.
+
+The body of a PL/Python user-defined function is a Python script. When the 
function is called, its arguments are passed as elements of the array `args[]`. 
You can also pass named arguments as ordinary variables to the Python script. 
 
-Then, run the following SQL command:
+PL/Python function results are returned with a `return` statement, or a 
`yield` statement in the case of a result-set statement.
 
-```shell
-psql# CREATE LANGUAGE plpythonu;
+The following PL/Python function computes and returns the maximum of two 
integers:
+
+``` sql
+=> CREATE FUNCTION mypymax (a integer, b integer)
+ RETURNS integer
+   AS $$
+ if (a is None) or (b is None):
+   return None
+ if a > b:
+   return a
+ return b
+   $$ LANGUAGE plpythonu;
 ```
 
-Note that `plpythonu` is installed as an “untrusted” language, meaning it 
does not offer any way of restricting what users can do in it.
+To execute the `mypymax` function:
 
-To remove support for `plpythonu` from a database, run the following SQL 
command:
+``` sql
+=> SELECT mypymax(5, 7);
+ mypymax 
+-
+   7
+(1 row)
+```
+
+Adding the `STRICT` keyword to the `LANGUAGE` subclause instructs HAWQ to 
return null when any of the 

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796691#comment-15796691
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94511905
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
+PL/Python user-defined functions (UDFs) are registered at the database 
level. To create and run a PL/Python UDF on a database, you must register the 
PL/Python language with the database. 
+
+On every database to which you want to install and enable PL/Python:
+
+1. Connect to the database using the `psql` client:
+
+``` shell
+$ psql -d 
+```
+
+Replace \ with the name of the target database.
+
+2. Run the following SQL command to register the PL/Python procedural 
language; you must be a database superuser to register new languages:
+
+``` sql
+dbname=# CREATE LANGUAGE plpythonu;
+```
 
-To create and run a PL/Python user-defined function (UDF) in a database, 
you must register the PL/Python language with the database. On every database 
where you want to install and enable PL/Python, connect to the database using 
the `psql` client.
+**Note**: `plpythonu` is installed as an *untrusted* language; it 
offers no way of restricting what you can program in UDFs created with the 
language.
 
--- End diff --

Given the definition, it seems like HAWQ treats plpythonu as a "trusted" 
language.  Meaning, it's permitted to do what it likes.  I'd expect an 
untrusted language to have restrictions applied to it.


> PL/Python doc page - validate and enhance info
> --
>
> Key: HAWQ-1216
> URL: https://issues.apache.org/jira/browse/HAWQ-1216
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> PL/Python docs section:
> - review and validate current content
> - update module installation instructions - need to be root to install 
> modules to system python dirs
> - make sure examples run and are relevant
> - develop new examples as appropriate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796694#comment-15796694
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94511080
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
--- End diff --

Change "when compiling" -> "when you compile"


> PL/Python doc page - validate and enhance info
> --
>
> Key: HAWQ-1216
> URL: https://issues.apache.org/jira/browse/HAWQ-1216
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> PL/Python docs section:
> - review and validate current content
> - update module installation instructions - need to be root to install 
> modules to system python dirs
> - make sure examples run and are relevant
> - develop new examples as appropriate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796693#comment-15796693
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94512620
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
+PL/Python user-defined functions (UDFs) are registered at the database 
level. To create and run a PL/Python UDF on a database, you must register the 
PL/Python language with the database. 
+
+On every database to which you want to install and enable PL/Python:
+
+1. Connect to the database using the `psql` client:
+
+``` shell
+$ psql -d 
+```
+
+Replace \ with the name of the target database.
+
+2. Run the following SQL command to register the PL/Python procedural 
language; you must be a database superuser to register new languages:
+
+``` sql
+dbname=# CREATE LANGUAGE plpythonu;
+```
 
-To create and run a PL/Python user-defined function (UDF) in a database, 
you must register the PL/Python language with the database. On every database 
where you want to install and enable PL/Python, connect to the database using 
the `psql` client.
+**Note**: `plpythonu` is installed as an *untrusted* language; it 
offers no way of restricting what you can program in UDFs created with the 
language.
 
-```shell
-$ psql -d 
+To remove support for `plpythonu` from a database, run the following SQL 
command; you must be a database superuser to remove a registered procedural 
language:
+
+``` sql
+dbname=# DROP LANGUAGE plpythonu;
 ```
 
-Replace \ with the name of the target database.
+## Developing Functions with PL/Python 
+
+PL/Python functions are defined using the standard SQL [CREATE 
FUNCTION](../reference/sql/CREATE-FUNCTION.html) syntax.
+
+The body of a PL/Python user-defined function is a Python script. When the 
function is called, its arguments are passed as elements of the array `args[]`. 
You can also pass named arguments as ordinary variables to the Python script. 
 
-Then, run the following SQL command:
+PL/Python function results are returned with a `return` statement, or a 
`yield` statement in the case of a result-set statement.
 
-```shell
-psql# CREATE LANGUAGE plpythonu;
+The following PL/Python function computes and returns the maximum of two 
integers:
+
+``` sql
+=> CREATE FUNCTION mypymax (a integer, b integer)
+ RETURNS integer
+   AS $$
+ if (a is None) or (b is None):
+   return None
+ if a > b:
+   return a
+ return b
+   $$ LANGUAGE plpythonu;
 ```
 
-Note that `plpythonu` is installed as an “untrusted” language, meaning it 
does not offer any way of restricting what users can do in it.
+To execute the `mypymax` function:
 
-To remove support for `plpythonu` from a database, run the following SQL 
command:
+``` sql
+=> SELECT mypymax(5, 7);
+ mypymax 
+-
+   7
+(1 row)
+```
+
+Adding the `STRICT` keyword to the `LANGUAGE` subclause instructs HAWQ to 
return null when any of the 

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796690#comment-15796690
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94511655
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
+PL/Python user-defined functions (UDFs) are registered at the database 
level. To create and run a PL/Python UDF on a database, you must register the 
PL/Python language with the database. 
+
+On every database to which you want to install and enable PL/Python:
+
+1. Connect to the database using the `psql` client:
+
+``` shell
+$ psql -d 
+```
+
+Replace \ with the name of the target database.
+
+2. Run the following SQL command to register the PL/Python procedural 
language; you must be a database superuser to register new languages:
+
--- End diff --

It seems like the superuser restriction would be better to call out in step 
1, where you can specify a user credential.


> PL/Python doc page - validate and enhance info
> --
>
> Key: HAWQ-1216
> URL: https://issues.apache.org/jira/browse/HAWQ-1216
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> PL/Python docs section:
> - review and validate current content
> - update module installation instructions - need to be root to install 
> modules to system python dirs
> - make sure examples run and are relevant
> - develop new examples as appropriate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1216) PL/Python doc page - validate and enhance info

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796692#comment-15796692
 ] 

ASF GitHub Bot commented on HAWQ-1216:
--

Github user dyozie commented on a diff in the pull request:

https://github.com/apache/incubator-hawq-docs/pull/77#discussion_r94511497
  
--- Diff: plext/using_plpython.html.md.erb ---
@@ -2,374 +2,608 @@
 title: Using PL/Python in HAWQ
 ---
 
-This section contains an overview of the HAWQ PL/Python language extension.
+This section provides an overview of the HAWQ PL/Python procedural 
language extension.
 
 ## About HAWQ PL/Python 
 
-PL/Python is a loadable procedural language. With the HAWQ PL/Python 
extension, you can write HAWQ user-defined functions in Python that take 
advantage of Python features and modules to quickly build robust database 
applications.
+PL/Python is embedded in your HAWQ product distribution or within your 
HAWQ build if you chose to enable it as a build option. 
+
+With the HAWQ PL/Python extension, you can write user-defined functions in 
Python that take advantage of Python features and modules, enabling you to 
quickly build robust HAWQ database applications.
 
 HAWQ uses the system Python installation.
 
 ### HAWQ PL/Python Limitations 
 
-- HAWQ does not support PL/Python triggers.
+- HAWQ does not support PL/Python trigger functions.
 - PL/Python is available only as a HAWQ untrusted language.
  
 ## Enabling and Removing PL/Python Support 
 
-To use PL/Python in HAWQ, you must either use a pre-compiled version of 
HAWQ that includes PL/Python or specify PL/Python as a build option when 
compiling HAWQ.
+To use PL/Python in HAWQ, you must either install a binary version of HAWQ 
that includes PL/Python or specify PL/Python as a build option when compiling 
HAWQ from source.
+
+PL/Python user-defined functions (UDFs) are registered at the database 
level. To create and run a PL/Python UDF on a database, you must register the 
PL/Python language with the database. 
--- End diff --

Rearrange the second sentence to something like:  You must register the 
PL/Python language with a database, before you can create and run a PL/Python 
UDF on that database.


> PL/Python doc page - validate and enhance info
> --
>
> Key: HAWQ-1216
> URL: https://issues.apache.org/jira/browse/HAWQ-1216
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: David Yozie
>Priority: Minor
>
> PL/Python docs section:
> - review and validate current content
> - update module installation instructions - need to be root to install 
> modules to system python dirs
> - make sure examples run and are relevant
> - develop new examples as appropriate



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-902) remove greenplum database references from HAWQ components

[Lisa Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lisa Owen updated HAWQ-902:
---
Component/s: Command Line Tools

> remove greenplum database references from HAWQ components
> -
>
> Key: HAWQ-902
> URL: https://issues.apache.org/jira/browse/HAWQ-902
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Affects Versions: 2.0.0.0-incubating, 2.1.0.0-incubating
>Reporter: Lisa Owen
>Assignee: Lei Chang
>Priority: Minor
> Fix For: backlog
>
>
> several gpdb(green plum database) references exist in HAWQ.  these include, 
> but are not limited to,:
> * $GPHOME
> * /usr/local/hawq/greenplum_path.sh
> * gp* named admin and client utilities 
> * gp* GUCs
> users who are not familiar with HAWQ's history may be confused by these 
> references.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1250) Create HAWQ-specific html docs

[David Yozie (JIRA)]

David Yozie created HAWQ-1250:
-

 Summary: Create HAWQ-specific html docs
 Key: HAWQ-1250
 URL: https://issues.apache.org/jira/browse/HAWQ-1250
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Documentation
Reporter: David Yozie
Assignee: David Yozie
 Fix For: 2.0.0.0-incubating


The HAWQ site is currently pointing to the HDP end-user docs.  Change this to 
host Apache HAWQ-specific docs directly off of apache.org for each new release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #1071: HAWQ-1242. hawq-site.xml default content has wro...

[jiny2]

Github user jiny2 commented on the issue:

https://github.com/apache/incubator-hawq/pull/1071
  
@huor @stanlyxiang @paul-guo- 

@stanlyxiang  is right, the correct approach is to fix guc.c as all 
external scripts and documentation are using hawq_* names.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94381769
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with 
other containers
+LOCAL := 
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all: 
+   @echo " Usage:"
+   @echo "To setup a build and test environment: make run"
+   @echo "To start all containers:   make start"
+   @echo "To stop all containers:make stop"
+   @echo "To remove hdfs containers: make clean"
+   @echo "To remove all containers:  make 
distclean"
+   @echo ""
+   @echo "To build images locally:   make build"
--- End diff --

What is the cleanup command for "make build"?  (make distclean or make 
clean?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94380992
  
--- Diff: contrib/hawq-docker/README.md ---
@@ -0,0 +1,96 @@
+# hawq-docker
+
+hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker 
images and scripts to help developers of Apache HAWQ to setup building and 
testing environment with docker.
+
+Both CentOS 7 and CentOS 6 are supported.
+Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to 
switch between CentOS 7 and CentOS 6.
+
+Take CentOS 7 as an example below.
+
+# Install docker
+* following the instructions to install docker.
+https://docs.docker.com/
+
+# Setup build and test environment
+* clone this repository
+```
+git clone https://github.com/guofengrichard/hawq-docker.git .
+```
+* Get the docker images
+```
+  cd hawq-docker
+  make pull (recommended)
+OR
+  make build
+``` 
+* setup a 5 nodes virtual cluster for Apache HAWQ build and test.
+```
+make run
+```
+Now let's have a look about what we creted.
+```
+[root@localhost hawq-docker]# docker ps -a
+CONTAINER IDIMAGE  COMMAND
CREATED STATUS  PORTS   NAMES
+382b2b3360d1hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode3
+86513c331d45hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode2
+c0ab10e46e4ahawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-datanode1
+e27beea63953hawq/hawq-test:centos7   "entrypoint.sh bash"   2 
minutes ago   Up 2 minutescentos7-namenode
+1f986959bd04hawq/hawq-dev:centos7"/bin/true"2 
minutes ago   Created centos7-data
+```
+**centos7-data** is a data container and mounted to /data directory on all 
other containers to provide a shared storage for the cluster. 
+
+# Build and Test Apache HAWQ
+* attach to namenode
+```
+docker exec -it centos7-namenode bash
+```
+* check if HDFS working well
+```
+sudo -u hdfs hdfs dfsadmin -report
+```
+* clone Apache HAWQ code to /data direcotry
+```
+git clone https://github.com/apache/incubator-hawq.git /data/hawq
+```
+* build Apache HAWQ
+```
+cd /data/hawq
+./configure --prefix=/data/hawq-dev
+make
+make install
+```
+(When you are using CentOS 6, run command `scl enable devtoolset-2 bash` 
before
+configuring hawq and run command `exit` after installing hawq.) 
--- End diff --

Why "exit" command is needed? exit the container?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94379844
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with 
other containers
+LOCAL := 
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all: 
+   @echo " Usage:"
+   @echo "To setup a build and test environment: make run"
+   @echo "To start all containers:   make start"
+   @echo "To stop all containers:make stop"
+   @echo "To remove hdfs containers: make clean"
+   @echo "To remove all containers:  make 
distclean"
+   @echo ""
+   @echo "To build images locally:   make build"
+   @echo "To pull latest images: make pull"
--- End diff --

I guess we do not promise we put the latest image on docker hub?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94381463
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
--- End diff --

Please add License header into all files which could add.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94380425
  
--- Diff: contrib/hawq-docker/README.md ---
@@ -0,0 +1,96 @@
+# hawq-docker
+
+hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker 
images and scripts to help developers of Apache HAWQ to setup building and 
testing environment with docker.
+
+Both CentOS 7 and CentOS 6 are supported.
+Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to 
switch between CentOS 7 and CentOS 6.
+
+Take CentOS 7 as an example below.
+
+# Install docker
+* following the instructions to install docker.
+https://docs.docker.com/
+
+# Setup build and test environment
+* clone this repository
+```
+git clone https://github.com/guofengrichard/hawq-docker.git .
+```
+* Get the docker images
+```
+  cd hawq-docker
+  make pull (recommended)
+OR
+  make build
--- End diff --

We need to add doc to tell users the difference. (pull vs build)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[paul-guo-]

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1073#discussion_r94379343
  
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
--- End diff --

What is the difference between NDATANODES and CUR_DATANODE?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1003) Implement batched ACL check through Ranger.

[Lili Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794682#comment-15794682
 ] 

Lili Ma commented on HAWQ-1003:
---

A query is usually composed of multiple ACL Check requests, for example, if 
insert into an empty table, a series queries for analyze will generate. If we 
can assemble all the requests into one, we will reduce the cost added on Ranger 
ACL.


> Implement batched ACL check through Ranger.
> ---
>
> Key: HAWQ-1003
> URL: https://issues.apache.org/jira/browse/HAWQ-1003
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Core
>Reporter: Lili Ma
>Assignee: hongwu
> Fix For: backlog
>
>
> Implement enhanced hawq ACL check through Ranger, which means, if a query 
> contains several tables, we can combine the multiple table request together, 
> to send just one REST request to Ranger REST API Server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1249) Don't do ACL checks on segments

[Chunling Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang reassigned HAWQ-1249:
---

Assignee: Chunling Wang  (was: Ed Espino)

> Don't do ACL checks on segments
> ---
>
> Key: HAWQ-1249
> URL: https://issues.apache.org/jira/browse/HAWQ-1249
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Chunling Wang
>Assignee: Chunling Wang
> Fix For: backlog
>
>
> HAWQ does ACL checks on segments, which we think is not necessary for QE 
> because there is no catalog data on segments. Even a hacker can connect to a 
> segdb with GP_ROLE_EXECUTE, he can not do any queries while he can do on 
> Greenplum for there is catalog data on segments. Further more, in ranger 
> checks, if all segments do same checks as master with RPS, it costs a lot and 
> effects the performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1249) Don't do ACL checks on segments

[Chunling Wang (JIRA)]

Chunling Wang created HAWQ-1249:
---

 Summary: Don't do ACL checks on segments
 Key: HAWQ-1249
 URL: https://issues.apache.org/jira/browse/HAWQ-1249
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Chunling Wang
Assignee: Ed Espino


HAWQ does ACL checks on segments, which we think is not necessary for QE 
because there is no catalog data on segments. Even a hacker can connect to a 
segdb with GP_ROLE_EXECUTE, he can not do any queries while he can do on 
Greenplum for there is catalog data on segments. Further more, in ranger 
checks, if all segments do same checks as master with RPS, it costs a lot and 
effects the performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1246) Add generation of RequestID, ClientIP, queryContext(SQL Statement) in HAWQ , and encapsulate these contents to JSON request to RPS

[Lili Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lili Ma updated HAWQ-1246:
--
Issue Type: Sub-task  (was: Bug)
Parent: HAWQ-256

> Add generation of RequestID, ClientIP, queryContext(SQL Statement) in HAWQ , 
> and encapsulate these contents to JSON request to RPS
> --
>
> Key: HAWQ-1246
> URL: https://issues.apache.org/jira/browse/HAWQ-1246
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Xiang Sheng
>Assignee: Xiang Sheng
> Fix For: 2.2.0.0-incubating
>
>
> These informations should be generated and encapsulate them to the full json 
> request. 
> Currently they are hardcoded.
> {code}
> json_object *jreqid = json_object_new_string("1");
> json_object_object_add(jrequest, "requestId", jreqid);
> json_object *jclientip = json_object_new_string("123.0.0.21");
> json_object_object_add(jrequest, "clientIp", jclientip);
> json_object *jcontext = json_object_new_string("SELECT * FROM DDD");
> json_object_object_add(jrequest, "context", jcontext);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1246) Add generation of RequestID, ClientIP, queryContext(SQL Statement) in HAWQ , and encapsulate these contents to JSON request to RPS

[Lili Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794657#comment-15794657
 ] 

Lili Ma commented on HAWQ-1246:
---

Current Request ID is managed per session level, and now we print it out in 
HAWQ master log.
```
2017-01-03 17:38:34.775632 
CST,"malili","postgres",p54608,th2056364032,"[local]",,2017-01-03 17:38:30 
CST,3586,con14,cmd2,seg-1,,,x3586,sx1,"LOG","0","Send JSON request to 
Ranger: { ""requestId"": ""8"", ""user"": ""malili"", ""clientIp"": 
""127.0.0.1"", ""context"": ""SELECT d.datname as \""Name\"",\n   
pg_catalog.pg_get_userbyid(d.datdba) as \""Owner\"",\n   
pg_catalog.pg_encoding_to_char(d.encoding) as \""Encoding\"",\n   
pg_catalog.array_to_string(d.datacl, E'\\n') AS \""Access privileges\""\nFROM 
pg_catalog.pg_database d\nWHERE d.datname <> 'hcatalog'\nORDER BY 1;"", 
""access"": [ { ""resource"": { ""database"": ""postgres"", ""schema"": 
""pg_catalog"", ""function"": ""pg_encoding_to_char"" }, ""privileges"": [ 
""EXECUTE"" ] } ] }",,"SELECT d.datname as ""Name"",
   pg_catalog.pg_get_userbyid(d.datdba) as ""Owner"",
   pg_catalog.pg_encoding_to_char(d.encoding) as ""Encoding"",
   pg_catalog.array_to_string(d.datacl, E'\n') AS ""Access privileges""
FROM pg_catalog.pg_database d
WHERE d.datname <> 'hcatalog'
ORDER BY 1;",0,,"rangerrest.c",391,
```

Note that it's session level.  We can use this information to detect what's a 
query is composed of.

> Add generation of RequestID, ClientIP, queryContext(SQL Statement) in HAWQ , 
> and encapsulate these contents to JSON request to RPS
> --
>
> Key: HAWQ-1246
> URL: https://issues.apache.org/jira/browse/HAWQ-1246
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Security
>Reporter: Xiang Sheng
>Assignee: Xiang Sheng
> Fix For: 2.2.0.0-incubating
>
>
> These informations should be generated and encapsulate them to the full json 
> request. 
> Currently they are hardcoded.
> {code}
> json_object *jreqid = json_object_new_string("1");
> json_object_object_add(jrequest, "requestId", jreqid);
> json_object *jclientip = json_object_new_string("123.0.0.21");
> json_object_object_add(jrequest, "clientIp", jclientip);
> json_object *jcontext = json_object_new_string("SELECT * FROM DDD");
> json_object_object_add(jrequest, "context", jcontext);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1200) Incremental make install.

[Paul Guo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794615#comment-15794615
 ] 

Paul Guo commented on HAWQ-1200:


Just saw it. It is an interesting idea. I have two concerns, 
1) I guess Makefiles have been with proper dependencies so that non-modified 
files will not be copied, else that is a Makefile bug.
2) Actually our install script install-sh supports similar idea via the option 
"-C". The difference is that is it uses cmp.
rsync is a bit heavy for comparison since it seems that it calculates using 
hash digest and allows shift.

I just quickly hacked src/backend/Makefile.global and test install performance 
on my virtual machine,

#INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c
INSTALL= $(SHELL) $(top_srcdir)/config/install-sh -C

No obvious improvement in my system. Actually the difference will depend on the 
cpu and io performance. Since the comparison solution will
introduce additional cpu cost for calculation and io cost for original file 
read. If write is slow (e.g. slow hard disk or through network), the solution
will be good, else no.

Anyone could easily customize it via setting the environment variable,
CUSTOM_INSTALL

> Incremental make install.
> -
>
> Key: HAWQ-1200
> URL: https://issues.apache.org/jira/browse/HAWQ-1200
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Reporter: hongwu
>Assignee: Lei Chang
>
> Current make install process cp all copy files from source directory to 
> install prefix, this is time-consuming. We can optimize it with rsync instead 
> of cp. It could improve the development efficiency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...

[guofengrichard]

GitHub user guofengrichard opened a pull request:

https://github.com/apache/incubator-hawq/pull/1073

HAWQ-1248. Merge Dockerfiles for HAWQ Dev into HAWQ code base.

This commit is to merge Dockerfiles into HAWQ code base. The Dockerfiles 
are used to create docker images, which provide an out-of-box way for 
developers to setup build and test environment for HAWQ.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guofengrichard/incubator-hawq hawq-docker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1073.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1073


commit e33743e12fc20cdd7496fb40ee5a3778786ee78e
Author: Richard Guo 
Date:   2017-01-03T09:12:59Z

HAWQ-1248. Merge Dockerfiles for HAWQ Dev into HAWQ code base.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---