[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-trafodion/pull/546


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[robertamarton]

Github user robertamarton commented on a diff in the pull request:

https://github.com/apache/incubator-trafodion/pull/546#discussion_r68089548
  
--- Diff: install/installer/traf_secure_setup ---
@@ -0,0 +1,342 @@
+#!/bin/bash
+
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+
+# KDC attributes
+MAX_LIFETIME="24hours"
+RENEW_LIFETIME="7days"
+
+# LDAP attributes
+LDAP_AUTH_FILE="traf_authentication_config_${HOSTNAME}"
+
+HOST_NAME=`hostname -f`
+
+# These differ depending on the distribution
+if [[ $HADOOP_TYPE == "cloudera" ]]; then
+  TRAF_KEYTAB_DIR='/etc/trafodion'
+  TRAF_KEYTAB="trafodion.keytab"
+  HBASE_KEYTAB=`sudo find /var/run/cloudera-scm-agent/process/ -name 
hbase.keytab | grep 'hbase-MASTER' | head -n 1`
+  HDFS_KEYTAB=`sudo find /var/run/cloudera-scm-agent/process/ -name 
hdfs.keytab | grep 'hdfs-NAMENODE/' | head -n 1`
+else
+  TRAF_KEYTAB_DIR='/etc/security/keytabs'
+  TRAF_KEYTAB="trafodion.service.keytab"
+  HBASE_KEYTAB='/etc/security/keytabs/hbase.service.keytab'
+  HDFS_KEYTAB='/etc/security/keytabs/hdfs.headless.keytab'
+fi
+
+#==
+#  Setup Trafodion environment for secure Hadoop
+#==
+
+LOCAL_WORKDIR="$( cd "$( dirname "$0" )" && pwd )"
+
+TRAF_CONFIG="/etc/trafodion/trafodion_config"
+LOCAL_SECURE_CONFIG="$LOCAL_WORKDIR/trafodion_secure_config"
+rm $LOCAL_SECURE_CONFIG  2>/dev/null
+
+
+#==
+# Kerberos enabled
+
+echo "***INFO: Starting Trafodion security configuration setup"
+
+# Check to see if kerberos is enabled in Hadoop
+cat /etc/hadoop/conf/core-site.xml | while read a; do
--- End diff --

Most definitely, was not aware of this environment variable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[hegdean]

Github user hegdean commented on a diff in the pull request:

https://github.com/apache/incubator-trafodion/pull/546#discussion_r68082081
  
--- Diff: install/installer/traf_secure_setup ---
@@ -0,0 +1,342 @@
+#!/bin/bash
+
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+
+# KDC attributes
+MAX_LIFETIME="24hours"
+RENEW_LIFETIME="7days"
+
+# LDAP attributes
+LDAP_AUTH_FILE="traf_authentication_config_${HOSTNAME}"
+
+HOST_NAME=`hostname -f`
+
+# These differ depending on the distribution
+if [[ $HADOOP_TYPE == "cloudera" ]]; then
+  TRAF_KEYTAB_DIR='/etc/trafodion'
+  TRAF_KEYTAB="trafodion.keytab"
+  HBASE_KEYTAB=`sudo find /var/run/cloudera-scm-agent/process/ -name 
hbase.keytab | grep 'hbase-MASTER' | head -n 1`
+  HDFS_KEYTAB=`sudo find /var/run/cloudera-scm-agent/process/ -name 
hdfs.keytab | grep 'hdfs-NAMENODE/' | head -n 1`
+else
+  TRAF_KEYTAB_DIR='/etc/security/keytabs'
+  TRAF_KEYTAB="trafodion.service.keytab"
+  HBASE_KEYTAB='/etc/security/keytabs/hbase.service.keytab'
+  HDFS_KEYTAB='/etc/security/keytabs/hdfs.headless.keytab'
+fi
+
+#==
+#  Setup Trafodion environment for secure Hadoop
+#==
+
+LOCAL_WORKDIR="$( cd "$( dirname "$0" )" && pwd )"
+
+TRAF_CONFIG="/etc/trafodion/trafodion_config"
+LOCAL_SECURE_CONFIG="$LOCAL_WORKDIR/trafodion_secure_config"
+rm $LOCAL_SECURE_CONFIG  2>/dev/null
+
+
+#==
+# Kerberos enabled
+
+echo "***INFO: Starting Trafodion security configuration setup"
+
+# Check to see if kerberos is enabled in Hadoop
+cat /etc/hadoop/conf/core-site.xml | while read a; do
--- End diff --

Use HADOOP_CNF_DIR environment variable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[robertamarton]

Github user robertamarton commented on a diff in the pull request:

https://github.com/apache/incubator-trafodion/pull/546#discussion_r67192402
  
--- Diff: core/sqf/sql/scripts/sqstart ---
@@ -220,6 +220,31 @@ function SQCheckOrphanProcesses {
 
 }
 
+function checkKerberos {
+
+   # Check to see if kerberos is enabled in Hadoop
+   cat /etc/hadoop/conf/core-site.xml | while read a; do
+ found=`echo $a | grep "hadoop.security.authentication" | wc -l`
+ if [[ $found -eq 1 ]]; then
+   read b
+   enabled=`echo $b | grep kerberos | wc -l`
--- End diff --

I want to improve the check to see if Kerberos is enabled in general.  
There is the issue you mention above and in a customer environment, core-site 
may not be in the default location (take install_local_hadoop for instance).   
With a different solution, I hope to avoid scanning core-site.   I did write up 
a JIRA to improve this (TRAFODION-2048).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[DaveBirdsall]

Github user DaveBirdsall commented on a diff in the pull request:

https://github.com/apache/incubator-trafodion/pull/546#discussion_r67187426
  
--- Diff: core/sqf/sql/scripts/sqstart ---
@@ -220,6 +220,31 @@ function SQCheckOrphanProcesses {
 
 }
 
+function checkKerberos {
+
+   # Check to see if kerberos is enabled in Hadoop
+   cat /etc/hadoop/conf/core-site.xml | while read a; do
+ found=`echo $a | grep "hadoop.security.authentication" | wc -l`
+ if [[ $found -eq 1 ]]; then
+   read b
+   enabled=`echo $b | grep kerberos | wc -l`
--- End diff --

This code seems to assume that kerberos is on the very next line after 
"hadoop.security.authorization". Which is probably true but not necessarily. 
They could be on the same line, for example, or a comment line could occur 
between them. That said, without a full-blown XML parsing this is probably as 
good as you'll get.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[DaveBirdsall]

Github user DaveBirdsall commented on a diff in the pull request:

https://github.com/apache/incubator-trafodion/pull/546#discussion_r67188129
  
--- Diff: install/installer/traf_add_kerberos ---
@@ -0,0 +1,204 @@
+#!/bin/bash
+
+# @@@ START COPYRIGHT @@@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# @@@ END COPYRIGHT @@@
+

+#**
+#  Sets up Trafodion environment for security features:
+#Kerberos

+#**
+
+TRAF_CONFIG='/etc/trafodion/trafodion_config'
+source $TRAF_CONFIG
+HOST_NAME=`hostname -f`
+

+#==
+# set up kerberos stuff
+if [[ "$SECURE_HADOOP" == "N" ]]; then
+  echo "***INFO: KERBEROS not available, skipping to next step"
+  exit 0
+fi
+
+echo
+echo "**"
+echo " TRAFODION KERBEROS SETUP "
+echo "**"
+echo
+echo "***INFO: Running KERBEROS installation"
+
+# get realm from admin principal
+REALM=${ADMIN_PRINCIPAL#*"@"}
+TRAF_PRINCIPAL="$TRAF_USER/$HOST_NAME@REALM"
+HBASE_PRINCIPAL="$HBASE_USER/$HOST_NAME@$REALM"
+PDCP="pdcp -R ssh "
+
+# test KDC server connection - always ask for the admin password
+echo -n "Enter admin password:"
+read -s answer
+if [[ ! -z $answer ]]; then ADMIN_PASSWD=$answer; fi
+echo ""
+
+KADMIN_CMD="sudo kadmin -p $ADMIN_PRINCIPAL -w $ADMIN_PASSWD -s 
$KDC_SERVER -q"
+
+$KADMIN_CMD "listprincs" > /dev/null 2>&1
+if [[ $? -ne 0 ]]; then
+  echo "***ERROR: kadmin command failed to execute, verify that Kerberos 
is running, you can access it from the installation node, and that your 
password is valid"
+  ADMIN_PASSWD="***"
+  exit -1
+fi
+echo "***INFO: Connection to  KDC server successful" 
+
+# Make a directory to hold generated keytabs, ignore if already created
+mkdir -p $LOCAL_WORKDIR/keytabs 2>/dev/null
+
+echo "***INFO: Create principals and keytabs for $TRAF_USER" 
+for ITEM in $HADOOP_NODES; do
+  NODE=`ssh -q -n $ITEM sudo hostname -f`
+  PRINCIPAL_EXISTS=$( $KADMIN_CMD "listprincs" | grep 
"$TRAF_USER/$NODE@$REALM" | wc -l )
+  if [[ $PRINCIPAL_EXISTS -eq 1 ]]; then
+echo "***INFO: Principal $TRAF_USER/$NODE@$REALM exists, continuing" 
+  else
+# add the principal
+$KADMIN_CMD "addprinc -randkey $TRAF_USER/$NODE@$REALM" > /dev/null 
2>&1 
+if [[ $? -ne 0 ]]; then
+  echo "***ERROR: kadmin command failed to create principal, check KDC 
server status" 
+  ADMIN_PASSWD="***"
+  exit -1
+fi
+  fi
+
+  # Adjust principal's maxlife and maxrenewlife
+  echo "***INFO: Set max and renew life times for principal 
$TRAF_USER/$NODE@$REALM" 
+  $KADMIN_CMD "modprinc -maxlife $MAX_LIFETIME -maxrenewlife 
$RENEW_LIFETIME $TRAF_USER/$NODE@$REALM" > /dev/null 2>&1 
+  if [[ $? -ne 0 ]]; then
+echo "***ERROR: kadmin command failed to modify principal, check KDC 
server status" 
+ADMIN_PASSWD="***"
+exit -1
+  fi
+  
+  # Look in keytabs to see if keytab already exists, if so, then skip this 
step
+  # May want to supporting regeneration of keytabs at some point in time.
+  echo "***INFO: Create keytab $TRAF_KEYTAB for $NODE" 
+  if [[ -e $LOCAL_WORKDIR/keytabs/$NODE-$TRAF_KEYTAB ]]; then
+echo "***INFO: The keytab for $NODE exists, continuing" 
+  else
+echo "***INFO: Adding keytab for $NODE" 
+$KADMIN_CMD "ktadd -k $LOCAL_WORKDIR/$TRAF_KEYTAB 
$TRAF_USER/$NODE@$REALM"
+if [[ $? -ne 0 ]]; then
+  echo "***ERROR: failed to add keytab" 
+  ADMIN_PASSWD="***"
+  exit -1
+fi
+
+# Each node has its own principal and keytab.  The principal names 
have the node
+# name embedded but the keytab names are the same.  Save 

[GitHub] incubator-trafodion pull request #546: TRAFODION [109] Instrument Trafodion ...

[robertamarton]

GitHub user robertamarton opened a pull request:

https://github.com/apache/incubator-trafodion/pull/546

TRAFODION [109] Instrument Trafodion to work with Secure Hadoop (Kerberos)

The first phase of automation with Kerberos is ready including:

New secure install: Customer installs Trafodion on a cluster with secure 
Hadoop. 
There is no existing Trafodion installation.

Conversion from non-secure to secure: Customer has Trafodion installed on a 
non-secure Hadoop cluster. Customer enables Hadoop security on this 
existing 
cluster. Customer runs a Trafodion script to enable Hadoop security 
integration 
with Trafodion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertamarton/incubator-trafodion kerberos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-trafodion/pull/546.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #546


commit 9673059cfb03abd8aabf37be1b4936ee520a4129
Author: Roberta Marton 
Date:   2016-06-10T14:50:06Z

TRAFODION [109] Instrument Trafodion to work with Secure Hadoop (Kerberos)

The first phase of automation with Kerberos is ready.

The following use cases are supported

I.  New secure install: Customer installs Trafodion on a cluster with
 secure Hadoop. There is no existing Trafodion installation.

II. Conversion from non-secure to secure: Customer has Trafodion installed
on a non-secure Hadoop cluster. Customer enables Hadoop security on this
existing cluster. Customer runs a Trafodion script to enable Hadoop security
integration with Trafodion.

See the JIRA for details on the changes.

commit 18e404b3ca0936e6267c4006fd5676ee536a6944
Author: Roberta Marton 
Date:   2016-06-14T20:21:46Z

TRAFODION [109] Instrument Trafodion to work with Secure Hadoop (Kerberos)

Fixed issue where Jenkins installation was failing.
Fixed problem where Kerberos ticket was not being initialized on all nodes
without logging on.
Fixed an issue with uninstall

commit db8c8655d582302679498d43e142406346a5fdc5
Author: Roberta Marton 
Date:   2016-06-14T22:21:08Z

Merge branch 'master' into kerberos

Conflicts:
core/sqf/sql/scripts/sqcheck

commit ae6b0ab792ebf6789f9cb1ac852beff1e6066e9e
Author: Roberta Marton 
Date:   2016-06-14T23:05:41Z

Fix issue with trafodion_configuration_defaults




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---