[GitHub] incubator-hawq pull request #1203: HAWQ-1420: Enable build of Ranger plugin ...
Github user ljainpivotalio closed the pull request at: https://github.com/apache/incubator-hawq/pull/1203 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1215: HAWQ-1415. Set the default_value of JAVA_...
Github user ljainpivotalio closed the pull request at: https://github.com/apache/incubator-hawq/pull/1215 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1215: HAWQ-1415. Set the default_value of JAVA_...
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1215 HAWQ-1415. Set the default_value of JAVA_HOME for running RPS. Move v⦠â¦ar definition statement (no logic change) You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415-4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215 commit 0d0064b88cbc8e84f0eee96f2fa70c32b9a9ab85 Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-04-11T22:51:25Z HAWQ-1415. Set the default_value of JAVA_HOME for running RPS. Move var definition statement (no logic change) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1203: HAWQ-1420: Enable build of Ranger plugin open so...
Github user ljainpivotalio commented on the issue:
https://github.com/apache/incubator-hawq/pull/1203
```
mvn ${MVN_OPTS} -N ${BUILD_OPTS} install
[INFO] Creating spec file
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/SPECS/hawq-ranger-plugin.spec
[INFO] Building target platforms: noarch-redhat-linux
[INFO] Building for target noarch-redhat-linux
[INFO] Processing files: hawq-ranger-plugin-2.2.0.0-1.el6.noarch
[INFO] Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
[INFO] Checking for unpackaged file(s): /usr/lib/rpm/check-files
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/buildroot
[INFO] Wrote:
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/RPMS/noarch/hawq-ranger-plugin-2.2.0.0-1.el6.noarch.rpm
[INFO] Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.EplE5i
[WARNING] + umask 022
[WARNING] + cd /root/ranger-plugin/target/rpm/hawq-ranger-plugin/BUILD
[WARNING] + /bin/rm -rf
/root/ranger-plugin/target/rpm/hawq-ranger-plugin/buildroot
[WARNING] + exit 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 30.102 s
[INFO] Finished at: 2017-03-31T23:37:12+00:00
[INFO] Final Memory: 14M/95M
[INFO]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1203: Enable build of Ranger plugin open source RPM
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1203 @radarwave The open source build can override these values using maven command line: BUILD_OPTS="-Drelease.version=bin -Dbuild.suffix= -Dhawq.dep.name=apache-hawq -Ddestination.dir=/usr/local/apache-hawq" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1203: Enable build of Ranger plugin open source RPM
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1203 No. The dependency on hawq needs to change to apache-hawq or must be excluded in branch for the open source binary build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1206: Set the default_value of JAVA_HOME for ru...
Github user ljainpivotalio commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1206#discussion_r108581251
--- Diff: ranger-plugin/scripts/enable-ranger-plugin.sh ---
@@ -211,6 +211,28 @@ function update_ranger_url() {
echo "Updated POLICY_MGR_URL to ${policy_mgr_url} in ${prop_file}"
}
+function update_java_home() {
+ local jdk64="/usr/jdk64"
+ local java_sdk="/etc/alternatives/java_sdk"
+
+ if [[ -d ${jdk64} ]]; then
+local DIR_NAME=$(ls ${jdk64} | sort -r | head -1)
+if [[ ${DIR_NAME} ]]; then
+ JAVA_HOME_DIR="${jdk64}/${DIR_NAME}"
+fi
+ elif [[ -d ${java_sdk} ]]; then
+JAVA_HOME_DIR="${java_sdk}"
+ fi
+
+ if [[ ${JAVA_HOME_DIR} ]]; then
+local prop_file=$(dirname ${SCRIPT_DIR})/etc/rps.properties
+sed -i -e "s|/usr/java/default|${JAVA_HOME_DIR}|g" ${prop_file}
--- End diff --
Good catch. Forgot to commit the other 2 files :-(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1206: Set the default_value of JAVA_HOME for ru...
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1206 Set the default_value of JAVA_HOME for running RPS Third try ;-( You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1206.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1206 commit ddbd8dc516529621c2f672b47cef3ac189466fda Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-29T02:05:52Z Set the default_value of JAVA_HOME for running RPS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1205: Set the default_value of JAVA_HOME for ru...
Github user ljainpivotalio closed the pull request at: https://github.com/apache/incubator-hawq/pull/1205 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1204: Hawq 1415
Github user ljainpivotalio closed the pull request at: https://github.com/apache/incubator-hawq/pull/1204 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1204: Hawq 1415
GitHub user ljainpivotalio reopened a pull request: https://github.com/apache/incubator-hawq/pull/1204 Hawq 1415 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1204.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1204 commit 7de14557db7c881e61600b5965b293ebc5f292ba Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-29T01:47:20Z Set the default_value of JAVA_HOME for running RPS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1205: Set the default_value of JAVA_HOME for ru...
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1205 Set the default_value of JAVA_HOME for running RPS You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1205.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1205 commit 7de14557db7c881e61600b5965b293ebc5f292ba Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-29T01:47:20Z Set the default_value of JAVA_HOME for running RPS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1204: Hawq 1415
Github user ljainpivotalio closed the pull request at: https://github.com/apache/incubator-hawq/pull/1204 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1204: Hawq 1415
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1204 Hawq 1415 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1204.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1204 commit fe4af61c29d09eb4ace738fc9fea9df64b81622d Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-27T23:28:42Z [#142344489] Set the default_value of JAVA_HOME for running RPS commit 0e13334a75fdaf22239142f7199ca357cbedc69d Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-29T01:47:20Z Set the default_value of JAVA_HOME for running RPS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1203: Enable build of Ranger plugin open source...
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1203 Enable build of Ranger plugin open source RPM You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1420 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1203.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1203 commit 7f17fa1a6ad3f5230fd24ac14c86e31b82511eea Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-29T00:31:02Z Enable build of Ranger plugin open source RPM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1197: [HAWQ-1415] Set the default_value of JAVA...
GitHub user ljainpivotalio opened a pull request: https://github.com/apache/incubator-hawq/pull/1197 [HAWQ-1415] Set the default_value of JAVA_HOME for running RPS You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljainpivotalio/incubator-hawq HAWQ-1415 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1197.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1197 commit fe4af61c29d09eb4ace738fc9fea9df64b81622d Author: ljainpivotalio <lj...@pivotal.io> Date: 2017-03-27T23:28:42Z [#142344489] Set the default_value of JAVA_HOME for running RPS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1145: HAWQ-1353. Added SOLR properties to RPS audit co...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1145 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1132: HAWQ-1343. - Install pip and minor formatting ch...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1132 brew install python --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1130: HAWQ-1340. Remove unused file - tools/bin/disqua...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1130 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1123: HAWQ-1331. Update hawq and pxf versions
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1123 Please create 2.2.json in tools/bin/gppylib/data as a copy of 2.1.json --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1114: HAWQ-1320. Remove reference to PXF rpm version i...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1114 Please feel free to merge when done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1114: HAWQ-1320. Remove reference to PXF rpm ve...
Github user ljainpivotalio commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1114#discussion_r100613055 --- Diff: pxf/Makefile --- @@ -38,6 +38,10 @@ ifneq "$(VENDOR)" "" BUILD_PARAMS+= -Pvendor="$(VENDOR)" endif +ifneq "$(PXF_VERSION)" "" +BUILD_PARAMSi+= -Pversion="$(PXF_VERSION)" --- End diff -- There is an extra i after BUILD_PARAMS. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1104: HAWQ-1306. Removed links in Javadoc referencing ...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1104 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1101: HAWQ-1294. Removed ALL privilege from HAWQ Range...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1101 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1012: Revert "HAWQ-1141. Correct default values."
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/1012 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #941: HAWQ-1079. Fixed issue with symlink.
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/941 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #914: HAWQ-1045. RPM updates for side by side install/u...
Github user ljainpivotalio commented on the issue: https://github.com/apache/incubator-hawq/pull/914 pxf-json.rpm should be part of pxf-hdfs.rpm IMHO. LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...
Github user ljainpivotalio commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/857#discussion_r76278294
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+ HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+ },
+ HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+ }
+}
+
+
+class APIClient:
+ """
+ Class which interacts with Ambari Server API
+ """
+
+ # Base API URL points to localhost. This script is to be executed on the
Ambari Server
+ BASE_API_URL = 'http://localhost:8080/api/v1'
+
+ def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' +
self.password).replace('\n', '')
+
+ def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format:
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str
else None
+
+ def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+ status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+ if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+ elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start
ambari-server.')
+ else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+ def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else
response_json['items'][0]['Clusters']['cluster_name']
+
+ def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+ cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+ raise Exception('No Stack found to be installed on the cluster
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'],
stack_versi
[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...
Github user ljainpivotalio commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/857#discussion_r76278102
--- Diff: contrib/hawq-ambari-plugin/pom.xml ---
@@ -0,0 +1,125 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+ org.apache.hawq
+ hawq-ambari-plugin
+ 2.0.1
+ hawq-ambari-plugin
+ http://maven.apache.org
+
+
+1.0.0
+UTF-8
+${basedir}/target/resources-temp
+/var/lib/hawq
+${hawq.lib.dir}/staging
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-resources-plugin
+3.0.1
+
+
+
+none
+
+ copy-resources
+
+
+
+
+
+
+${basedir}/build.properties
+
+ ${resources.temp}
+
+
+ ${basedir}/src/main/resources/
+ true
+
+
+
+
+
+
+org.codehaus.mojo
+rpm-maven-plugin
+2.1.4
+
+
+
+none
+
+ rpm
+
+
+
+
+
+ org.apache.hawq
+
+HAWQ plugin contains Ambari's stack definition for HAWQ.
+When installed, Ambari will be able to support HAWQ as a
service.
+
+ x86_64
+ ${build_number}%{?dist}
+
+ambari-server = 2.2
+python = 2.6
+
+
+
+ ${hawq.lib.staging.dir}
+
+
+ ${resources.temp}/services
+
+
+
+
+ ${hawq.lib.dir}
+ 755
+
+
+ ${resources.temp}/utils
+
+
+
+
+
+
[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...
Github user ljainpivotalio commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/857#discussion_r76179421
--- Diff: contrib/hawq-ambari-plugin/README.md ---
@@ -0,0 +1,97 @@
+# HAWQ Ambari Plugin
+
+hawq-ambari-plugin helps users install HAWQ and PXF using Ambari.
+
+To ensure that Ambari recognizes HAWQ and PXF as services that can be
installed for a specific stack, the following manual steps are required:
+ * Add HAWQ and PXF metainfo.xml files (containing metadata about the
service) under the stack to be installed.
+ * Add repositories where HAWQ and PXF rpms reside, so that Ambari can use
it during installation. This requires updating repoinfo.xml under the stack
HAWQ and PXF is to be added.
+ * If a stack is already installed using Ambari, add repositories to the
existing stack using the Ambari REST API.
+
+
+## Source Code
+Source code directory structure of the hawq-ambari-plugin is as follows:
+```
+hawq-ambari-plugin
++-- README.md
++-- build.properties
++-- pom.xml
++-- src
++-- main
++-- resources
++-- services
+¦ +-- HAWQ
+¦ ¦ +-- metainfo.xml
+¦ +-- PXF
+¦ +-- metainfo.xml
++-- utils
++-- add-hawq.py
+```
+
+### build.properties
+[build.properties](build.properties) contains properties required for
building the plugin.
+
+### metainfo.xml
+[metainfo.xml](src/main/resources/services/HAWQ/metainfo.xml) contains the
metadata about the service. The metainfo.xml specifies that the service
definition is to be inherited from Ambari common-services. HAWQ and PXF
common-services code can be found under [Apache Ambari
repository](https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/).
+
+### add-hawq.py
+[add-hawq.py](src/main/resources/utils/add-hawq.py) deploys HAWQ and PXF
metainfo.xml files under the stack and adds the repositories to Ambari.
+
+
+## Building the plugin
+***Build Environment***: centos6 is the typical operating system used for
building.
+
+Properties specified in the [build.properties](build.properties) file:
+
+| Property | Description | Value |
+| --- | --- | --- |
+| hawq.release.version | Release version of HAWQ | 2.0.1 |
+| hawq.common.services.version | HAWQ common services code in Ambari to be
inherited | 2.0.0 |
+| pxf.release.version | Release version of PXF | 3.0.1 |
+| pxf.common.services.version | PXF common services code in Ambari to be
inherited | 3.0.0 |
+| hawq.repo.prefix | Repository name for HAWQ core repository | hawq |
+| hawq.addons.repo.prefix | Repository name for HAWQ Add Ons repository |
hawq-add-ons |
+| repository.version | Repository Version to be used in repository
information | 2.0.1.0 |
+| default.stack | Default stack under which, metainfo.xml and repositories
have to be added | HDP-2.4 |
+
+To build the rpm for hawq-ambari-plugin, change the
[build.properties](build.properties) file with the required parameters and run
```mvn install``` command under hawq-ambari-plugin directory:
+```
+$ pwd
+incubator-hawq/contrib/hawq-ambari-plugin
+$ mvn install
--- End diff --
mvn clean resources:copy-resources rpm:rpm -Dbuild_number="${BUILD_NUMBER}"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #857: HAWQ-1013. Move HAWQ Ambari plugin to Apac...
Github user ljainpivotalio commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/857#discussion_r76179141
--- Diff: contrib/hawq-ambari-plugin/src/main/resources/utils/add-hawq.py
---
@@ -0,0 +1,498 @@
+#!/usr/bin/env python
+
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import base64
+import getpass
+import json
+import os
+import shutil
+import socket
+import urllib2
+import xml.etree.ElementTree as ET
+from optparse import OptionParser
+
+DEFAULT_STACK = '${default.stack}'
+SUPPORTED_OS_LIST = ['redhat6']
+HAWQ_LIB_STAGING_DIR = '${hawq.lib.staging.dir}'
+REPO_VERSION = '${repository.version}'
+HAWQ_REPO = '${hawq.repo.prefix}'
+HAWQ_ADD_ONS_REPO = '${hawq.addons.repo.prefix}'
+
+REPO_INFO = {
+ HAWQ_REPO: {
+'repoid': '-'.join([HAWQ_REPO, REPO_VERSION]),
+'input_param': '--hawqrepo',
+'optional': False
+ },
+ HAWQ_ADD_ONS_REPO: {
+'repoid': '-'.join([HAWQ_ADD_ONS_REPO, REPO_VERSION]),
+'input_param': '--addonsrepo',
+'optional': True
+ }
+}
+
+
+class APIClient:
+ """
+ Class which interacts with Ambari Server API
+ """
+
+ # Base API URL points to localhost. This script is to be executed on the
Ambari Server
+ BASE_API_URL = 'http://localhost:8080/api/v1'
+
+ def __init__(self, user, password):
+self.user = user
+self.password = password
+self.encoded_credentials = base64.encodestring(self.user + ':' +
self.password).replace('\n', '')
+
+ def __request(self, method, url_path, headers=None, data=None):
+"""
+Creates API requests and packages response into the following format:
(response code, response body in json object)
+"""
+headers = headers if headers is not None else {}
+headers['Authorization'] = 'Basic {0}'.format(self.encoded_credentials)
+
+req = urllib2.Request(self.BASE_API_URL + url_path, data, headers)
+req.get_method = lambda: method
+response = urllib2.urlopen(req)
+response_str = response.read()
+
+return response.getcode(), json.loads(response_str) if response_str
else None
+
+ def verify_api_reachable(self):
+"""
+Returns true if Ambari Server is reachable through API
+"""
+try:
+ status_code, _ = self.__request('GET', '/stacks')
+except Exception as e:
+ if type(e) == urllib2.HTTPError and e.code == 403:
+raise Exception('Invalid username and/or password.')
+ elif type(e) == urllib2.URLError:
+raise Exception('Ambari-server is not running. Please start
ambari-server.')
+ else:
+raise Exception('Unable to connect to Ambari Server.\n' + str(e))
+
+ def get_cluster_name(self):
+"""
+Returns the name of the installed cluster
+"""
+_, response_json = self.__request('GET', '/clusters')
+return None if len(response_json['items']) == 0 else
response_json['items'][0]['Clusters']['cluster_name']
+
+ def get_stack_info(self, cluster_name):
+"""
+Returns stack information (stack name, stack version, repository
version) of stack installed on cluster
+"""
+_, response_json = self.__request('GET',
+
'/clusters/{0}/stack_versions?ClusterStackVersions/state.matches(CURRENT)'.format(
+ cluster_name))
+if 'items' not in response_json or len(response_json['items']) == 0:
+ raise Exception('No Stack found to be installed on the cluster
{0}'.format(cluster_name))
+stack_versions = response_json['items'][0]['ClusterStackVersions']
+return stack_versions['stack'], stack_versions['version'],
stack_versi
