date:20160825

[jira] [Commented] (HAWQ-984) hawq config is too slow.

2016-08-25 Thread Paul Guo (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438486#comment-15438486
 ] 

Paul Guo commented on HAWQ-984:
---

1) gpscp
It still uses popen2.Popen4(). This function introduces millions of useless 
close() sys call finally in above test command. Using subprocess.Popen()  
without close_fds as an alternative easily resolves this.

2) gppylib/commands/base.py

def __init__(self,name,pool,timeout=5):

The worker thread will block at most 5 seconds in each loop (Queue.get()) to 
fetch potential commands even we have known that there will be no more commands 
to run for some threads. This really does not make sense since some idle 
threads will block for 5 seconds also before exiting.

Setting timeout to zero will make python code spin. I tested a small timeout 
value e.g. 0.1s and it works fine. It seems that  0.1 is a good timeout 
candidate.

> hawq config is too slow.
> 
>
> Key: HAWQ-984
> URL: https://issues.apache.org/jira/browse/HAWQ-984
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Paul Guo
>Assignee: Lei Chang
> Fix For: 2.0.1.0-incubating
>
>
> I tried to set a simple guc value via "hawq config" on my centos
> virtual system, but it spends >6 seconds. I know "hawq config" just
> simple scp-update xml files to various nodes. This should have
> been very fast if network situation is fine (My test hawq system is all-in-one
> test environment so network latency is not an issue.)
> I expect this is done on my test system (just one seg node) in < 1 second.
> But actually the time is:
> $ time hawq config -c lc_messages -v en_US.UTF-8
> GUC lc_messages already exist in hawq-site.xml
> Update it with value: en_US.UTF-8
> GUC   : lc_messages
> Value : en_US.UTF-8
> real  0m6.363s
> user  0m0.943s
> sys   0m1.258s
> I quickly monitored the processes when running "hawq config",
> it seems that the scp command finishes very early and adding
> print debug code in python file gpscp shows that the code in it
> finishes also very early.
> I'm not sure if anyone sees that long like my case (my system
> is probably quite mangled after dev and test). I roughly traced
> the python stack. I suspect there is something wrong in python
> or we are not using related module well or we should replace modules.
> I just checked "hawq config" but it is possible other CLI suffers from this 
> also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-984) hawq config is too slow.

2016-08-25 Thread Paul Guo (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo closed HAWQ-984.
-
   Resolution: Fixed
 Assignee: Paul Guo  (was: Lei Chang)
Fix Version/s: (was: backlog)
   2.0.1.0-incubating

> hawq config is too slow.
> 
>
> Key: HAWQ-984
> URL: https://issues.apache.org/jira/browse/HAWQ-984
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.0.1.0-incubating
>
>
> I tried to set a simple guc value via "hawq config" on my centos
> virtual system, but it spends >6 seconds. I know "hawq config" just
> simple scp-update xml files to various nodes. This should have
> been very fast if network situation is fine (My test hawq system is all-in-one
> test environment so network latency is not an issue.)
> I expect this is done on my test system (just one seg node) in < 1 second.
> But actually the time is:
> $ time hawq config -c lc_messages -v en_US.UTF-8
> GUC lc_messages already exist in hawq-site.xml
> Update it with value: en_US.UTF-8
> GUC   : lc_messages
> Value : en_US.UTF-8
> real  0m6.363s
> user  0m0.943s
> sys   0m1.258s
> I quickly monitored the processes when running "hawq config",
> it seems that the scp command finishes very early and adding
> print debug code in python file gpscp shows that the code in it
> finishes also very early.
> I'm not sure if anyone sees that long like my case (my system
> is probably quite mangled after dev and test). I roughly traced
> the python stack. I suspect there is something wrong in python
> or we are not using related module well or we should replace modules.
> I just checked "hawq config" but it is possible other CLI suffers from this 
> also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #863: HAWQ-1021. Need to log for some local_ssh functio...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/863
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #856: HAWQ-984. hawq config is too slow.

2016-08-25 Thread paul-guo-

Github user paul-guo- closed the pull request at:

https://github.com/apache/incubator-hawq/pull/856


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #862: HAWQ-900. Add dependency in PL/R rpm build...

2016-08-25 Thread paul-guo-

Github user paul-guo- closed the pull request at:

https://github.com/apache/incubator-hawq/pull/862


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #856: HAWQ-984. hawq config is too slow.

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/856
  
Merged. Closing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-1011) Check whether the table to be registered is existed

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu resolved HAWQ-1011.
--
Resolution: Implemented

> Check whether the table to be registered is existed
> ---
>
> Key: HAWQ-1011
> URL: https://issues.apache.org/jira/browse/HAWQ-1011
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> Check whether the table to be registered is existed or not. If it is, print a 
> error message and then exit. This is a feature for hawq register to make sure 
> the right name from user input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #859: HAWQ-1011. Check whether the table to be r...

2016-08-25 Thread xunzhang

Github user xunzhang closed the pull request at:

https://github.com/apache/incubator-hawq/pull/859


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #859: HAWQ-1011. Check whether the table to be register...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/859
  
Merged into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (HAWQ-838) include an alternative for paramiko python module

2016-08-25 Thread Radar Lei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-838.

Resolution: Fixed

> include an alternative for paramiko python module
> -
>
> Key: HAWQ-838
> URL: https://issues.apache.org/jira/browse/HAWQ-838
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Minor
> Fix For: 2.0.1.0-incubating
>
>
> Since paramiko's license is conflict with Apach HAWQ's license, so we can't 
> deliver it with HAWQ package. So users need to install it before installing 
> hawq.
> To save user's time to install paramiko manually, we can find a good 
> replacement which don't have license conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-1008) Remove legacy options for hawq command line tools

2016-08-25 Thread Radar Lei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1008.
-
Resolution: Fixed

> Remove legacy options for hawq command line tools
> -
>
> Key: HAWQ-1008
> URL: https://issues.apache.org/jira/browse/HAWQ-1008
> Project: Apache HAWQ
>  Issue Type: Task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: Radar Lei
>Assignee: Radar Lei
>Priority: Minor
> Fix For: 2.0.1.0-incubating
>
>
> There are some unused/legacy command line tool options, we should remove them.
> See:
> hawq state: 
>   -d MASTER_DIR, --datadir=MASTER_DIR
> HAWQ Master data directory.
>   --hawqhome=HAWQ_HOME  HAWQ Master data directory.
> hawq config: 
> -d HAWQ_HOME  HAWQ home directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-1009) Remove requirement of environment value 'MASTER_DATA_DIRECTORY'

2016-08-25 Thread Radar Lei (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1009.
-
Resolution: Fixed

> Remove requirement of environment value 'MASTER_DATA_DIRECTORY'
> ---
>
> Key: HAWQ-1009
> URL: https://issues.apache.org/jira/browse/HAWQ-1009
> Project: Apache HAWQ
>  Issue Type: Task
>Reporter: Radar Lei
>Assignee: Radar Lei
> Fix For: 2.0.1.0-incubating
>
>
> In old HAWQ command line tools, some tools will require 
> 'MASTER_DATA_DIRECTORY' environment to be set. 
> Since now we define 'MASTER_DATA_DIRECTORY' in hawq-site.xml, we do not need 
> to set it anymore. So we'd better remove these requirements to avoid user 
> confusion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #852: HAWQ-1008. Remove legacy options for hawq ...

2016-08-25 Thread radarwave

Github user radarwave closed the pull request at:

https://github.com/apache/incubator-hawq/pull/852


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #859: HAWQ-1011. Check whether the table to be register...

2016-08-25 Thread ictmalili

Github user ictmalili commented on the issue:

https://github.com/apache/incubator-hawq/pull/859
  
LGTM. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #859: HAWQ-1011. Check whether the table to be register...

2016-08-25 Thread radarwave

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/859
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #859: HAWQ-1011. Check whether the table to be register...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/859
  
also cc @radarwave 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #864: HAWQ-1009 Remove requirement of environmen...

2016-08-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/864


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #859: HAWQ-1011. Check whether the table to be r...

2016-08-25 Thread ictmalili

Github user ictmalili commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/859#discussion_r76351833
  
--- Diff: tools/bin/hawqregister ---
@@ -131,6 +130,18 @@ def option_parser_yml(yml_file):
 
 def create_table(dburl, tablename, schema_info, fmt, distrbution_policy, 
file_locations):
 try:
+query = "select count(*) from pg_class where relname = '%s'" % 
tablename.split('.')[-1]
--- End diff --

Maybe we need to consider the upper/lower cases of tablename.  All the 
table name stored in HAWQ are converted to lower case. So I think we need to 
convert tablename to lower case and then do comparison. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1025) Check the consistency of AO/Parquet_FileLocations.Files.size attribute in extracted yaml file and the actual file size in HDFS.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1025:


Assignee: hongwu  (was: Lei Chang)

> Check the consistency of AO/Parquet_FileLocations.Files.size attribute in 
> extracted yaml file and the actual file size in HDFS.
> ---
>
> Key: HAWQ-1025
> URL: https://issues.apache.org/jira/browse/HAWQ-1025
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1024) Rollback if hawq register failed in process

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1024:


Assignee: hongwu  (was: Lei Chang)

> Rollback if hawq register failed in process
> ---
>
> Key: HAWQ-1024
> URL: https://issues.apache.org/jira/browse/HAWQ-1024
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1024) Rollback if hawq register failed in process

2016-08-25 Thread hongwu (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438238#comment-15438238
 ] 

hongwu commented on HAWQ-1024:
--

In the second use of hawq register, if hawq register fail in process, we should 
drop the created table.

> Rollback if hawq register failed in process
> ---
>
> Key: HAWQ-1024
> URL: https://issues.apache.org/jira/browse/HAWQ-1024
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1024) Rollback if hawq register failed in process

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1024:
-
Affects Version/s: 2.0.1.0-incubating

> Rollback if hawq register failed in process
> ---
>
> Key: HAWQ-1024
> URL: https://issues.apache.org/jira/browse/HAWQ-1024
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1023) Incorrect usage of java.lang.String.replaceAll

2016-08-25 Thread hongwu (JIRA)

hongwu created HAWQ-1023:


 Summary: Incorrect usage of java.lang.String.replaceAll
 Key: HAWQ-1023
 URL: https://issues.apache.org/jira/browse/HAWQ-1023
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: hongwu
Assignee: Lei Chang


Incorrect usage for java.lang.String.replaceAll generate useless calls:
https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPath.java#L51

https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPoint.java#L45



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1024) Rollback if hawq register failed in process

2016-08-25 Thread hongwu (JIRA)

hongwu created HAWQ-1024:


 Summary: Rollback if hawq register failed in process
 Key: HAWQ-1024
 URL: https://issues.apache.org/jira/browse/HAWQ-1024
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Command Line Tools
Reporter: hongwu
Assignee: Lei Chang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #866: HAWQ-1013. Add Copyright to hawq-ambari-pl...

2016-08-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/866


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #866: HAWQ-1013. Add Copyright to hawq-ambari-pl...

2016-08-25 Thread mithmatt

GitHub user mithmatt opened a pull request:

https://github.com/apache/incubator-hawq/pull/866

HAWQ-1013. Add Copyright to hawq-ambari-plugin



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mithmatt/incubator-hawq add-copyright-field

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/866.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #866






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt closed the pull request at:

https://github.com/apache/incubator-hawq/pull/857


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76317991
  
--- Diff: contrib/hawq-ambari-plugin/build.properties ---
@@ -0,0 +1,8 @@
+hawq.release.version=2.0.1
+hawq.common.services.version=2.0.0
+pxf.release.version=3.0.1
+pxf.common.services.version=3.0.0
+hawq.repo.prefix=hawq
+hawq.addons.repo.prefix=hawq-add-ons
+repository.version=2.0.1.0
+default.stack=HDP-2.4
--- End diff --

Nothing to do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76317914
  
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py 
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+  HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+  },
+  HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+  }
+}
+
+
+class APIClient:
+  """
+  Class which interacts with Ambari Server API
+  """
+
+  # Base API URL points to localhost. This script is to be executed on the 
Ambari Server
+  BASE_API_URL = 'http://localhost:8080/api/v1'
+
+  def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' + 
self.password).replace('\n', '')
+
+  def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format: 
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str 
else None
+
+  def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+  status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+  if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+  elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start 
ambari-server.')
+  else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+  def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else 
response_json['items'][0]['Clusters']['cluster_name']
+
+  def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository 
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+  
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+  cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+  raise Exception('No Stack found to be installed on the cluster 
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'], 
stack_versions['repository_version']
+
+  def get_existing_repository_info(self, stack_name, stack_version, 
repository_version):
+"""
+Returns existing repo information for a given stack
+"""
+

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76317948
  
--- Diff: contrib/hawq-ambari-plugin/pom.xml ---
@@ -0,0 +1,125 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+  4.0.0
+  org.apache.hawq
+  hawq-ambari-plugin
+  2.0.1
+  hawq-ambari-plugin
+  http://maven.apache.org
+
+  
+1.0.0
+UTF-8
+${basedir}/target/resources-temp
+/var/lib/hawq
+${hawq.lib.dir}/staging
+  
+
+  
+
+
+  
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.1
+
+
+  
+none
+
+  copy-resources
+
+  
+
+
+
+  
+${basedir}/build.properties
+  
+  ${resources.temp}
+  
+
+  ${basedir}/src/main/resources/
+  true
+
+  
+
+  
+
+  
+org.codehaus.mojo
+rpm-maven-plugin
+2.1.4
+
+
+  
+none
+
+  rpm
+
+  
+
+
+
+  org.apache.hawq
+  
+HAWQ plugin contains Ambari's stack definition for HAWQ.
+When installed, Ambari will be able to support HAWQ as a 
service.
+  
+  x86_64
+  ${build_number}%{?dist}
+  
+ambari-server = 2.2
+python = 2.6
+  
+  
+
+  ${hawq.lib.staging.dir}
+  
+
+  ${resources.temp}/services
+
+  
+
+
+  ${hawq.lib.dir}
+  755
+  
+
+  ${resources.temp}/utils
+
+  
+
+  
+  
+

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76317925
  
--- Diff: contrib/hawq-ambari-plugin/README.md ---
@@ -0,0 +1,97 @@
+# HAWQ Ambari Plugin
+
+hawq-ambari-plugin helps users install HAWQ and PXF using Ambari.
+
+To ensure that Ambari recognizes HAWQ and PXF as services that can be 
installed for a specific stack, the following manual steps are required:
+ * Add HAWQ and PXF metainfo.xml files (containing metadata about the 
service) under the stack to be installed.
+ * Add repositories where HAWQ and PXF rpms reside, so that Ambari can use 
it during installation. This requires updating repoinfo.xml under the stack 
HAWQ and PXF is to be added.
+ * If a stack is already installed using Ambari, add repositories to the 
existing stack using the Ambari REST API.
+
+
+## Source Code
+Source code directory structure of the hawq-ambari-plugin is as follows:
+```
+hawq-ambari-plugin
++-- README.md
++-- build.properties
++-- pom.xml
++-- src
++-- main
++-- resources
++-- services
+Â¦   +-- HAWQ
+Â¦   Â¦   +-- metainfo.xml
+Â¦   +-- PXF
+Â¦   +-- metainfo.xml
++-- utils
++-- add-hawq.py
+```
+
+### build.properties
+[build.properties](build.properties) contains properties required for 
building the plugin.
+
+### metainfo.xml
+[metainfo.xml](src/main/resources/services/HAWQ/metainfo.xml) contains the 
metadata about the service. The metainfo.xml specifies that the service 
definition is to be inherited from Ambari common-services. HAWQ and PXF 
common-services code can be found under [Apache Ambari 
repository](https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/).
+
+### add-hawq.py
+[add-hawq.py](src/main/resources/utils/add-hawq.py) deploys HAWQ and PXF 
metainfo.xml files under the stack and adds the repositories to Ambari.
+
+
+## Building the plugin
+***Build Environment***: centos6 is the typical operating system used for 
building.
+
+Properties specified in the [build.properties](build.properties) file:
+
+| Property | Description | Value |
+| --- | --- | --- |
+| hawq.release.version | Release version of HAWQ | 2.0.1 |
+| hawq.common.services.version | HAWQ common services code in Ambari to be 
inherited | 2.0.0 |
+| pxf.release.version | Release version of PXF | 3.0.1 |
+| pxf.common.services.version | PXF common services code in Ambari to be 
inherited | 3.0.0 |
+| hawq.repo.prefix | Repository name for HAWQ core repository  | hawq |
+| hawq.addons.repo.prefix | Repository name for HAWQ Add Ons repository  | 
hawq-add-ons |
+| repository.version | Repository Version to be used in repository 
information | 2.0.1.0 |
+| default.stack | Default stack under which, metainfo.xml and repositories 
have to be added | HDP-2.4 |
+
+To build the rpm for hawq-ambari-plugin, change the 
[build.properties](build.properties) file with the required parameters and run 
```mvn install``` command under hawq-ambari-plugin directory:
+```
+$ pwd
+incubator-hawq/contrib/hawq-ambari-plugin
+$ mvn install
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...

2016-08-25 Thread oarap

Github user oarap commented on the issue:

https://github.com/apache/incubator-hawq/pull/839
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #838: HAWQ-934. Populate canSetTag of PlannedStmt from ...

2016-08-25 Thread oarap

Github user oarap commented on the issue:

https://github.com/apache/incubator-hawq/pull/838
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #857: HAWQ-1013. Move HAWQ Ambari plugin to Apache HAWQ

2016-08-25 Thread vVineet

Github user vVineet commented on the issue:

https://github.com/apache/incubator-hawq/pull/857
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76301165
  
--- Diff: contrib/hawq-ambari-plugin/build.properties ---
@@ -0,0 +1,8 @@
+hawq.release.version=2.0.1
+hawq.common.services.version=2.0.0
+pxf.release.version=3.0.1
+pxf.common.services.version=3.0.0
+hawq.repo.prefix=hawq
+hawq.addons.repo.prefix=hawq-add-ons
+repository.version=2.0.1.0
+default.stack=HDP-2.4
--- End diff --

Keeping build.properties for ease of use with CI builds


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76301092
  
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py 
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+  HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+  },
+  HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+  }
+}
+
+
+class APIClient:
+  """
+  Class which interacts with Ambari Server API
+  """
+
+  # Base API URL points to localhost. This script is to be executed on the 
Ambari Server
+  BASE_API_URL = 'http://localhost:8080/api/v1'
+
+  def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' + 
self.password).replace('\n', '')
+
+  def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format: 
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str 
else None
+
+  def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+  status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+  if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+  elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start 
ambari-server.')
+  else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+  def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else 
response_json['items'][0]['Clusters']['cluster_name']
+
+  def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository 
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+  
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+  cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+  raise Exception('No Stack found to be installed on the cluster 
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'], 
stack_versions['repository_version']
+
+  def get_existing_repository_info(self, stack_name, stack_version, 
repository_version):
+"""
+Returns existing repo information for a given stack
+"""
+

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76289077
  
--- Diff: contrib/hawq-ambari-plugin/pom.xml ---
@@ -0,0 +1,125 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+  4.0.0
+  org.apache.hawq
+  hawq-ambari-plugin
+  2.0.1
+  hawq-ambari-plugin
+  http://maven.apache.org
+
+  
+1.0.0
+UTF-8
+${basedir}/target/resources-temp
+/var/lib/hawq
+${hawq.lib.dir}/staging
+  
+
+  
+
+
+  
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.1
+
+
+  
+none
+
+  copy-resources
+
+  
+
+
+
+  
+${basedir}/build.properties
+  
+  ${resources.temp}
+  
+
+  ${basedir}/src/main/resources/
+  true
+
+  
+
+  
+
+  
+org.codehaus.mojo
+rpm-maven-plugin
+2.1.4
+
+
+  
+none
+
+  rpm
+
+  
+
+
+
+  org.apache.hawq
+  
+HAWQ plugin contains Ambari's stack definition for HAWQ.
+When installed, Ambari will be able to support HAWQ as a 
service.
+  
+  x86_64
+  ${build_number}%{?dist}
+  
+ambari-server = 2.2
+python = 2.6
+  
+  
+
+  ${hawq.lib.staging.dir}
+  
+
+  ${resources.temp}/services
+
+  
+
+
+  ${hawq.lib.dir}
+  755
+  
+
+  ${resources.temp}/utils
+
+  
+
+  
+  
+

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread ljainpivotalio

Github user ljainpivotalio commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76278294
  
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py 
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+  HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+  },
+  HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+  }
+}
+
+
+class APIClient:
+  """
+  Class which interacts with Ambari Server API
+  """
+
+  # Base API URL points to localhost. This script is to be executed on the 
Ambari Server
+  BASE_API_URL = 'http://localhost:8080/api/v1'
+
+  def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' + 
self.password).replace('\n', '')
+
+  def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format: 
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str 
else None
+
+  def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+  status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+  if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+  elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start 
ambari-server.')
+  else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+  def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else 
response_json['items'][0]['Clusters']['cluster_name']
+
+  def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository 
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+  
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+  cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+  raise Exception('No Stack found to be installed on the cluster 
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'], 
stack_versions['repository_version']
+
+  def get_existing_repository_info(self, stack_name, stack_version, 
repository_version):
+"""
+Returns existing repo information for a given stack
+"""

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread ljainpivotalio

Github user ljainpivotalio commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76278102
  
--- Diff: contrib/hawq-ambari-plugin/pom.xml ---
@@ -0,0 +1,125 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+  4.0.0
+  org.apache.hawq
+  hawq-ambari-plugin
+  2.0.1
+  hawq-ambari-plugin
+  http://maven.apache.org
+
+  
+1.0.0
+UTF-8
+${basedir}/target/resources-temp
+/var/lib/hawq
+${hawq.lib.dir}/staging
+  
+
+  
+
+
+  
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.1
+
+
+  
+none
+
+  copy-resources
+
+  
+
+
+
+  
+${basedir}/build.properties
+  
+  ${resources.temp}
+  
+
+  ${basedir}/src/main/resources/
+  true
+
+  
+
+  
+
+  
+org.codehaus.mojo
+rpm-maven-plugin
+2.1.4
+
+
+  
+none
+
+  rpm
+
+  
+
+
+
+  org.apache.hawq
+  
+HAWQ plugin contains Ambari's stack definition for HAWQ.
+When installed, Ambari will be able to support HAWQ as a 
service.
+  
+  x86_64
+  ${build_number}%{?dist}
+  
+ambari-server = 2.2
+python = 2.6
+  
+  
+
+  ${hawq.lib.staging.dir}
+  
+
+  ${resources.temp}/services
+
+  
+
+
+  ${hawq.lib.dir}
+  755
+  
+
+  ${resources.temp}/utils
+
+  
+
+  
+  
+

[jira] [Resolved] (HAWQ-1022) Fix Command::getCommandOutput bug: using reference to temporary variable in stack.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu resolved HAWQ-1022.
--
Resolution: Fixed

> Fix Command::getCommandOutput bug: using reference to temporary variable in 
> stack.
> --
>
> Key: HAWQ-1022
> URL: https://issues.apache.org/jira/browse/HAWQ-1022
> Project: Apache HAWQ
>  Issue Type: Bug
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #865: HAWQ-1022. Bugfix for getCommandOutput: us...

2016-08-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/865


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #865: HAWQ-1022. Bugfix for getCommandOutput: using ref...

2016-08-25 Thread huor

Github user huor commented on the issue:

https://github.com/apache/incubator-hawq/pull/865
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1022) Fix Command::getCommandOutput bug: using reference to temporary variable in stack.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1022:
-
Fix Version/s: 2.0.1.0-incubating

> Fix Command::getCommandOutput bug: using reference to temporary variable in 
> stack.
> --
>
> Key: HAWQ-1022
> URL: https://issues.apache.org/jira/browse/HAWQ-1022
> Project: Apache HAWQ
>  Issue Type: Bug
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1022) Fix Command::getCommandOutput bug: using reference to temporary variable in stack.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1022:
-
Affects Version/s: 2.0.1.0-incubating

> Fix Command::getCommandOutput bug: using reference to temporary variable in 
> stack.
> --
>
> Key: HAWQ-1022
> URL: https://issues.apache.org/jira/browse/HAWQ-1022
> Project: Apache HAWQ
>  Issue Type: Bug
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1011) Check whether the table to be registered is existed

2016-08-25 Thread hongwu (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436683#comment-15436683
 ] 

hongwu commented on HAWQ-1011:
--

[~kdunn926], good advice! I think in our first 
implementation(v2.0.1.0-incubating), for simplicity, the default behaviour is 
fine. --force or --repair will be implemented as future options for hawq 
register. I will create another JIRA for that improvement in the future. Thanks 
again for the advice.

> Check whether the table to be registered is existed
> ---
>
> Key: HAWQ-1011
> URL: https://issues.apache.org/jira/browse/HAWQ-1011
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> Check whether the table to be registered is existed or not. If it is, print a 
> error message and then exit. This is a feature for hawq register to make sure 
> the right name from user input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #865: HAWQ-1022. Bugfix for getCommandOutput: using ref...

2016-08-25 Thread wcl14

Github user wcl14 commented on the issue:

https://github.com/apache/incubator-hawq/pull/865
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #865: HAWQ-1022. Bugfix for getCommandOutput: using ref...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/865
  
also cc @wcl14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #865: HAWQ-1022. Bugfix for getCommandOutput: us...

2016-08-25 Thread xunzhang

GitHub user xunzhang opened a pull request:

https://github.com/apache/incubator-hawq/pull/865

HAWQ-1022. Bugfix for getCommandOutput: using reference to temporary 
variable in stack address.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xunzhang/incubator-hawq HAWQ-1022

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/865.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #865


commit 35e7aa43ab9bd37fd194c6eb81821dc37c19319e
Author: xunzhang 
Date:   2016-08-25T10:24:52Z

HAWQ-1022. Bugfix for getCommandOutput: using reference to temporary 
variable in stack address.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #865: HAWQ-1022. Bugfix for getCommandOutput: using ref...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/865
  
cc @huor @paul-guo- 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1022) Fix Command::getCommandOutput bug: using reference to temporary variable in stack.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1022:


Assignee: hongwu  (was: Lei Chang)

> Fix Command::getCommandOutput bug: using reference to temporary variable in 
> stack.
> --
>
> Key: HAWQ-1022
> URL: https://issues.apache.org/jira/browse/HAWQ-1022
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: hongwu
>Assignee: hongwu
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1021) Need to log for some local_ssh function calls.

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1021:


Assignee: hongwu  (was: Lei Chang)

> Need to log for some local_ssh function calls.
> --
>
> Key: HAWQ-1021
> URL: https://issues.apache.org/jira/browse/HAWQ-1021
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: hongwu
>
> HAWQ management tools calls a lot of local_ssh() function to run some 
> external commands. The function is defined in hawqpylib/hawqlib.py. Many 
> callers does not set logger so we do not know any details about the command 
> running process. This is annoying when users/developers fail to run some 
> related commands and want to know the root cause quickly.
> Besides, there is two definitions of local_ssh(). Although they are not in 
> the same namespace, it is kind of annoying. We need to rename either one or 
> even both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1022) Fix Command::getCommandOutput bug: using reference to temporary variable in stack.

2016-08-25 Thread hongwu (JIRA)

hongwu created HAWQ-1022:


 Summary: Fix Command::getCommandOutput bug: using reference to 
temporary variable in stack.
 Key: HAWQ-1022
 URL: https://issues.apache.org/jira/browse/HAWQ-1022
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: hongwu
Assignee: Lei Chang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-969) Add getting configuration from HDFS and YARN

2016-08-25 Thread Chunling Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang closed HAWQ-969.
--

> Add getting configuration from HDFS and YARN
> 
>
> Key: HAWQ-969
> URL: https://issues.apache.org/jira/browse/HAWQ-969
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Chunling Wang
>Assignee: Chunling Wang
> Fix For: 2.0.1.0-incubating
>
>
> Add getting configuration from HDFS and YARN and writing xml file in 
> xml_parser.cpp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-1020) Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse

2016-08-25 Thread Chunling Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang closed HAWQ-1020.
---

> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse
> ---
>
> Key: HAWQ-1020
> URL: https://issues.apache.org/jira/browse/HAWQ-1020
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.0.1.0-incubating
>Reporter: Chunling Wang
>Assignee: Chunling Wang
> Fix For: 2.0.1.0-incubating
>
>
> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-969) Add getting configuration from HDFS and YARN

2016-08-25 Thread Chunling Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang resolved HAWQ-969.

Resolution: Fixed

> Add getting configuration from HDFS and YARN
> 
>
> Key: HAWQ-969
> URL: https://issues.apache.org/jira/browse/HAWQ-969
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Chunling Wang
>Assignee: Chunling Wang
> Fix For: 2.0.1.0-incubating
>
>
> Add getting configuration from HDFS and YARN and writing xml file in 
> xml_parser.cpp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #864: HAWQ-1009 Remove requirement of environment value...

2016-08-25 Thread xunzhang

Github user xunzhang commented on the issue:

https://github.com/apache/incubator-hawq/pull/864
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #863: HAWQ-1021. Need to log for some local_ssh functio...

2016-08-25 Thread radarwave

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/863
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #863: HAWQ-1021. Need to log for some local_ssh functio...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/863
  
@xunzhang @radarwave 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #807: HAWQ-897. Add feature test for create tabl...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/807#discussion_r76211338
  
--- Diff: src/test/feature/catalog/test_create_table.cpp ---
@@ -22,26 +22,26 @@ class TestCreateTable : public ::testing::Test {
 TEST_F(TestCreateTable, TestCreateTable1) {
   hawq::test::SQLUtility util;
   // prepare
-  util.execute("DROP TABLE IF EXISTS aggtest");
--- End diff --

I think jinyi's intent is to test "create/drop table".

Actually the constructor and destructor of class hawq::test::SQLUtility are 
exactly doing what you've suggested.

See src/test/feature/lib/sql_util.cpp for details.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #864: HAWQ-1009 Remove requirement of environment value...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/864
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #861: HAWQ-1001. Implement HAWQ user ACL check t...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/861#discussion_r76209892
  
--- Diff: src/backend/utils/misc/guc.c ---
@@ -349,6 +349,7 @@ bool
Debug_datumstream_write_use_small_initial_buffers = false;
 bool   gp_temporary_files_filespace_repair = false;
 bool   filesystem_support_truncate = true;
 bool   gp_allow_non_uniform_partitioning_ddl = true;
+bool  enable_ranger = false;
--- End diff --

ident


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #861: HAWQ-1001. Implement HAWQ user ACL check t...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/861#discussion_r76209911
  
--- Diff: src/backend/utils/misc/guc.c ---
@@ -4323,6 +4324,16 @@ static struct config_bool ConfigureNamesBool[] =
},
 
{
+{"enable_ranger", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
+ gettext_noop("support to using ranger to manage hawq privilege."),
--- End diff --

s/using/use/

s/ranger/Apache Ranger/ ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #861: HAWQ-1001. Implement HAWQ user ACL check through ...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/861
  
Whether need a test case for this code change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #863: HAWQ-1021. Need to log for some local_ssh ...

2016-08-25 Thread paul-guo-

GitHub user paul-guo- opened a pull request:

https://github.com/apache/incubator-hawq/pull/863

HAWQ-1021. Need to log for some local_ssh function calls.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-guo-/incubator-hawq python2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #863


commit 4cf7458785b7c10a2681ecbce300821f60e2bdb9
Author: Paul Guo 
Date:   2016-08-25T09:05:38Z

HAWQ-1021. Need to log for some local_ssh function calls.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-1021) Need to log for some local_ssh function calls.

2016-08-25 Thread Paul Guo (JIRA)

Paul Guo created HAWQ-1021:
--

 Summary: Need to log for some local_ssh function calls.
 Key: HAWQ-1021
 URL: https://issues.apache.org/jira/browse/HAWQ-1021
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Paul Guo
Assignee: Lei Chang


HAWQ management tools calls a lot of local_ssh() function to run some external 
commands. The function is defined in hawqpylib/hawqlib.py. Many callers does 
not set logger so we do not know any details about the command running process. 
This is annoying when users/developers fail to run some related commands and 
want to know the root cause quickly.

Besides, there is two definitions of local_ssh(). Although they are not in the 
same namespace, it is kind of annoying. We need to rename either one or even 
both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #862: HAWQ-900. Add dependency in PL/R rpm build spec f...

2016-08-25 Thread huor

Github user huor commented on the issue:

https://github.com/apache/incubator-hawq/pull/862
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #862: HAWQ-900. Add dependency in PL/R rpm build spec f...

2016-08-25 Thread radarwave

Github user radarwave commented on the issue:

https://github.com/apache/incubator-hawq/pull/862
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #862: HAWQ-900. Add dependency in PL/R rpm build...

2016-08-25 Thread paul-guo-

GitHub user paul-guo- opened a pull request:

https://github.com/apache/incubator-hawq/pull/862

HAWQ-900. Add dependency in PL/R rpm build spec file plr.spec

To build, r-core-devel is enough (no need of r-devel).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-guo-/incubator-hawq plr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/862.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #862


commit ffe8e486e55adc7a35f93d28e342e1f4bdbeaa99
Author: Paul Guo 
Date:   2016-07-06T10:09:42Z

HAWQ-900. Add dependency in PL/R rpm build spec file plr.spec




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #861: HAWQ-1001. Implement HAWQ user ACL check t...

2016-08-25 Thread zhangh43

GitHub user zhangh43 opened a pull request:

https://github.com/apache/incubator-hawq/pull/861

HAWQ-1001. Implement HAWQ user ACL check through Ranger.

Along with HAWQ-1002. Implement a switch in hawq-site.xml to configure 
whether use Ranger or not for ACL.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhangh43/incubator-hawq ranger

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/861.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #861


commit 6e1f7f8b2a93e38e206ddfca4bec30820097a3f3
Author: hzhang2 
Date:   2016-08-18T09:08:58Z

HAWQ-1002. Implement a switch in hawq-site.xml to configure whether use 
Ranger or not for ACL.

commit 7f1245477b80f3f1cf86947eb1401f57ac8f2938
Author: hzhang2 
Date:   2016-08-25T03:06:43Z

HAWQ-1001. Implement HAWQ user ACL check through Ranger.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1020) Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse

2016-08-25 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1020:
-
Affects Version/s: 2.0.1.0-incubating

> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse
> ---
>
> Key: HAWQ-1020
> URL: https://issues.apache.org/jira/browse/HAWQ-1020
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.0.1.0-incubating
>Reporter: Chunling Wang
>Assignee: Chunling Wang
>
> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #860: HAWQ-1020. Fix bugs to let feature tests TestComm...

2016-08-25 Thread paul-guo-

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/860
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76187867
  
--- Diff: contrib/hawq-ambari-plugin/build.properties ---
@@ -0,0 +1,8 @@
+hawq.release.version=2.0.1
+hawq.common.services.version=2.0.0
+pxf.release.version=3.0.1
+pxf.common.services.version=3.0.0
+hawq.repo.prefix=hawq
+hawq.addons.repo.prefix=hawq-add-ons
+repository.version=2.0.1.0
+default.stack=HDP-2.4
--- End diff --

There is no major problem with putting these parameters back in the pom.xml

I find it to be cleaner and better managed as a separate file. 

For the commercial release, a different set of build.properties can be 
used. It is easier to use a new build.properties file rather than updating the 
pom.xml or explicitly specify overrides for each parameter in the mvn command 
during the build process. 

Do you still want it back in the pom.xml?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #860: HAWQ-1020. Fix bugs to let feature tests T...

2016-08-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/860


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #860: HAWQ-1020. Fix bugs to let feature tests TestComm...

2016-08-25 Thread huor

Github user huor commented on the issue:

https://github.com/apache/incubator-hawq/pull/860
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76186961
  
--- Diff: contrib/hawq-ambari-plugin/README.md ---
@@ -0,0 +1,97 @@
+# HAWQ Ambari Plugin
+
+hawq-ambari-plugin helps users install HAWQ and PXF using Ambari.
+
+To ensure that Ambari recognizes HAWQ and PXF as services that can be 
installed for a specific stack, the following manual steps are required:
+ * Add HAWQ and PXF metainfo.xml files (containing metadata about the 
service) under the stack to be installed.
+ * Add repositories where HAWQ and PXF rpms reside, so that Ambari can use 
it during installation. This requires updating repoinfo.xml under the stack 
HAWQ and PXF is to be added.
+ * If a stack is already installed using Ambari, add repositories to the 
existing stack using the Ambari REST API.
+
+
+## Source Code
+Source code directory structure of the hawq-ambari-plugin is as follows:
+```
+hawq-ambari-plugin
++-- README.md
++-- build.properties
++-- pom.xml
++-- src
++-- main
++-- resources
++-- services
+Â¦   +-- HAWQ
+Â¦   Â¦   +-- metainfo.xml
+Â¦   +-- PXF
+Â¦   +-- metainfo.xml
++-- utils
++-- add-hawq.py
+```
+
+### build.properties
+[build.properties](build.properties) contains properties required for 
building the plugin.
+
+### metainfo.xml
+[metainfo.xml](src/main/resources/services/HAWQ/metainfo.xml) contains the 
metadata about the service. The metainfo.xml specifies that the service 
definition is to be inherited from Ambari common-services. HAWQ and PXF 
common-services code can be found under [Apache Ambari 
repository](https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/).
+
+### add-hawq.py
+[add-hawq.py](src/main/resources/utils/add-hawq.py) deploys HAWQ and PXF 
metainfo.xml files under the stack and adds the repositories to Ambari.
+
+
+## Building the plugin
+***Build Environment***: centos6 is the typical operating system used for 
building.
+
+Properties specified in the [build.properties](build.properties) file:
+
+| Property | Description | Value |
+| --- | --- | --- |
+| hawq.release.version | Release version of HAWQ | 2.0.1 |
+| hawq.common.services.version | HAWQ common services code in Ambari to be 
inherited | 2.0.0 |
+| pxf.release.version | Release version of PXF | 3.0.1 |
+| pxf.common.services.version | PXF common services code in Ambari to be 
inherited | 3.0.0 |
+| hawq.repo.prefix | Repository name for HAWQ core repository  | hawq |
+| hawq.addons.repo.prefix | Repository name for HAWQ Add Ons repository  | 
hawq-add-ons |
+| repository.version | Repository Version to be used in repository 
information | 2.0.1.0 |
+| default.stack | Default stack under which, metainfo.xml and repositories 
have to be added | HDP-2.4 |
+
+To build the rpm for hawq-ambari-plugin, change the 
[build.properties](build.properties) file with the required parameters and run 
```mvn install``` command under hawq-ambari-plugin directory:
+```
+$ pwd
+incubator-hawq/contrib/hawq-ambari-plugin
+$ mvn install
--- End diff --

Nice catch. Will update the build command.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76186913
  
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py 
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+  HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+  },
+  HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+  }
+}
+
+
+class APIClient:
+  """
+  Class which interacts with Ambari Server API
+  """
+
+  # Base API URL points to localhost. This script is to be executed on the 
Ambari Server
+  BASE_API_URL = 'http://localhost:8080/api/v1'
+
+  def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' + 
self.password).replace('\n', '')
+
+  def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format: 
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str 
else None
+
+  def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+  status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+  if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+  elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start 
ambari-server.')
+  else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+  def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else 
response_json['items'][0]['Clusters']['cluster_name']
+
+  def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository 
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+  
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+  cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+  raise Exception('No Stack found to be installed on the cluster 
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'], 
stack_versions['repository_version']
+
+  def get_existing_repository_info(self, stack_name, stack_version, 
repository_version):
+"""
+Returns existing repo information for a given stack
+"""
+

[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...

2016-08-25 Thread mithmatt

Github user mithmatt commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/857#discussion_r76186755
  
--- Diff: contrib/hawq-ambari-plugin/pom.xml ---
@@ -0,0 +1,125 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+  4.0.0
+  org.apache.hawq
+  hawq-ambari-plugin
+  2.0.1
+  hawq-ambari-plugin
+  http://maven.apache.org
+
+  
+1.0.0
+UTF-8
+${basedir}/target/resources-temp
+/var/lib/hawq
+${hawq.lib.dir}/staging
+  
+
+  
+
+
+  
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.1
+
+
+  
+none
+
+  copy-resources
+
+  
+
+
+
+  
+${basedir}/build.properties
+  
+  ${resources.temp}
+  
+
+  ${basedir}/src/main/resources/
+  true
+
+  
+
+  
+
+  
+org.codehaus.mojo
+rpm-maven-plugin
+2.1.4
+
+
+  
+none
+
+  rpm
+
+  
+
+
+
+  org.apache.hawq
+  
+HAWQ plugin contains Ambari's stack definition for HAWQ.
+When installed, Ambari will be able to support HAWQ as a 
service.
+  
+  x86_64
+  ${build_number}%{?dist}
+  
+ambari-server = 2.2
+python = 2.6
+  
+  
+
+  ${hawq.lib.staging.dir}
+  
+
+  ${resources.temp}/services
+
+  
+
+
+  ${hawq.lib.dir}
+  755
+  
+
+  ${resources.temp}/utils
+
+  
+
+  
+  
+

[GitHub] incubator-hawq pull request #860: HAWQ-1020. Fix bugs to let feature tests T...

2016-08-25 Thread huor

Github user huor commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/860#discussion_r76186124
  
--- Diff: src/test/feature/testlib/test_lib.cpp ---
@@ -64,6 +64,13 @@ TEST_F(TestCommonLib, TestHdfsConfig) {
   hc.isConfigKerberos();
   hc.isTruncate();
   std::string hadoopHome = hc.getHadoopHome();
+  /* chmod 777 $HADOOP_HOME/etc/hadoop/hdfs-site.xml */
--- End diff --

It would be better to comment like:
```
Grant access privilege for hdfs-site.xml
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #860: HAWQ-1020. Fix bugs to let feature tests T...

2016-08-25 Thread wcl14

GitHub user wcl14 opened a pull request:

https://github.com/apache/incubator-hawq/pull/860

HAWQ-1020. Fix bugs to let feature tests TestCommonLib.TestHdfsConfigâ¦

â¦ and TestCommonLib.TestYanConfig run in concourse

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wcl14/incubator-hawq HAWQ-1020

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #860


commit 1b2e004834ff2f71596ebd892008e6a63a70aaea
Author: Chunling Wang 
Date:   2016-08-25T06:11:16Z

HAWQ-1020. Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
TestCommonLib.TestYanConfig run in concourse




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (HAWQ-1020) Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse

2016-08-25 Thread Chunling Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang reassigned HAWQ-1020:
---

Assignee: Chunling Wang  (was: Jiali Yao)

> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse
> ---
>
> Key: HAWQ-1020
> URL: https://issues.apache.org/jira/browse/HAWQ-1020
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Tests
>Reporter: Chunling Wang
>Assignee: Chunling Wang
>
> Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
> TestCommonLib.TestYanConfig run in concourse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1020) Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse

2016-08-25 Thread Chunling Wang (JIRA)

Chunling Wang created HAWQ-1020:
---

 Summary: Fix bugs to let feature tests 
TestCommonLib.TestHdfsConfig and TestCommonLib.TestYanConfig run in concourse
 Key: HAWQ-1020
 URL: https://issues.apache.org/jira/browse/HAWQ-1020
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Tests
Reporter: Chunling Wang
Assignee: Jiali Yao


Fix bugs to let feature tests TestCommonLib.TestHdfsConfig and 
TestCommonLib.TestYanConfig run in concourse



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

2016-08-25 Thread wcl14

Github user wcl14 closed the pull request at:

https://github.com/apache/incubator-hawq/pull/828


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #807: HAWQ-897. Add feature test for create tabl...

2016-08-25 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/807#discussion_r76184425
  
--- Diff: src/test/feature/catalog/test_create_table.cpp ---
@@ -22,26 +22,26 @@ class TestCreateTable : public ::testing::Test {
 TEST_F(TestCreateTable, TestCreateTable1) {
   hawq::test::SQLUtility util;
   // prepare
-  util.execute("DROP TABLE IF EXISTS aggtest");
--- End diff --

The easier way to do this is creating a new schema `X`, set the default 
search_path to `X`. At the end of the test, drop the schema. In this way, you 
don't need to take care of dropping tables when you add new test cases. This is 
how GPDB is doing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

83 matches

Mail list logo