ndimiduk commented on a change in pull request #1620: URL: https://github.com/apache/hbase/pull/1620#discussion_r439096517
########## File path: dev-support/create-release/mac-sshd-gpg-agent/Dockerfile ########## @@ -0,0 +1,100 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Image for use on Mac boxes to get a gpg agent socket available Review comment: This is unique to mac as the host os? What about a linux host? ########## File path: dev-support/create-release/README.txt ########## @@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \ stable" $ sudo apt-get update $ sudo apt-get install -y docker-ce docker-ce-cli containerd.io -$ sudo usermod -a -G docker $USERID +# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/ +$ sudo usermod -aG docker $USER # LOGOUT and then LOGIN again so $USERID shows as part of docker group -# Copy up private key for $USERID export from laptop and import on gce. -$ gpg --import stack.duboce.net.asc -$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798 -$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400) -$ export PROJECT="${PROJECT:-hbase}" -$ git clone https://github.com/apache/${PROJECT}.git -$ cd "${PROJECT}" +# Test here by running docker's hello world as your build user +$ docker run hello-world + +# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM +# https://wiki.gnupg.org/AgentForwarding +# On the VM find out the location of the gpg agent socket and extra socket +$ gpgconf --list-dir agent-socket +/run/user/1000/gnupg/S.gpg-agent +$ gpgconf --list-dir agent-extra-socket +/run/user/1000/gnupg/S.gpg-agent.extra +# On the VM configure sshd to remove stale sockets +$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config' +$ sudo systemctl restart ssh +# logout of the VM + +# Do these steps on your local machine. +# Export your public key and copy it to the VM. +# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP) +$ gpg --export [email protected] > ~/gpg.example.apache.pub +$ scp ~/gpg.example.apache.pub example.gce.host: +# ssh into the VM while forwarding the remote gpg socket locations found above to your local +# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have +# your agent handle. Note that the gpg guide above can help you set this up in your ssh config +# rather than typing it in ssh like this every time. +$ ssh -i ~/.ssh/my_id \ + -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \ + -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \ + example.gce.host + +# now in an SSH session on the VM with the socket forwarding +# import your public key and test signing with the forwarding to your local agent. +$ gpg --no-autostart --import gpg.example.apache.pub +$ echo "foo" > foo.txt +$ gpg --no-autostart --detach --armor --sign foo.txt +$ gpg --no-autostart --verify foo.txt.asc + +# install git and clone the main project on the build machine +$ sudo apt-get install -y git +$ git clone https://github.com/apache/hbase.git Review comment: nit: should we point to gitbox instead? ########## File path: dev-support/create-release/release-util.sh ########## @@ -381,8 +383,6 @@ function configure_maven { <password>${env.ASF_PASSWORD}</password></server> <server><id>apache.releases.https</id><username>${env.ASF_USERNAME}</username> <password>${env.ASF_PASSWORD}</password></server> - <server><id>gpg.passphrase</id> - <passphrase>${env.GPG_PASSPHRASE}</passphrase></server> Review comment: 👍 ########## File path: dev-support/create-release/README.txt ########## @@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \ stable" $ sudo apt-get update $ sudo apt-get install -y docker-ce docker-ce-cli containerd.io -$ sudo usermod -a -G docker $USERID +# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/ +$ sudo usermod -aG docker $USER # LOGOUT and then LOGIN again so $USERID shows as part of docker group -# Copy up private key for $USERID export from laptop and import on gce. -$ gpg --import stack.duboce.net.asc -$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798 -$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400) -$ export PROJECT="${PROJECT:-hbase}" -$ git clone https://github.com/apache/${PROJECT}.git -$ cd "${PROJECT}" +# Test here by running docker's hello world as your build user +$ docker run hello-world + +# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM +# https://wiki.gnupg.org/AgentForwarding +# On the VM find out the location of the gpg agent socket and extra socket +$ gpgconf --list-dir agent-socket +/run/user/1000/gnupg/S.gpg-agent +$ gpgconf --list-dir agent-extra-socket +/run/user/1000/gnupg/S.gpg-agent.extra +# On the VM configure sshd to remove stale sockets +$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config' +$ sudo systemctl restart ssh +# logout of the VM + +# Do these steps on your local machine. +# Export your public key and copy it to the VM. +# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP) +$ gpg --export [email protected] > ~/gpg.example.apache.pub +$ scp ~/gpg.example.apache.pub example.gce.host: +# ssh into the VM while forwarding the remote gpg socket locations found above to your local +# gpg-agent's extra socket (this will restrict what commands the remote node is allowed to have +# your agent handle. Note that the gpg guide above can help you set this up in your ssh config +# rather than typing it in ssh like this every time. +$ ssh -i ~/.ssh/my_id \ + -R "/run/user/1000/gnupg/S.gpg-agent:$(gpgconf --list-dir agent-extra-socket)" \ + -R "/run/user/1000/gnupg/S.gpg-agent.extra:$(gpgconf --list-dir agent-extra-socket)" \ + example.gce.host + +# now in an SSH session on the VM with the socket forwarding +# import your public key and test signing with the forwarding to your local agent. +$ gpg --no-autostart --import gpg.example.apache.pub +$ echo "foo" > foo.txt +$ gpg --no-autostart --detach --armor --sign foo.txt +$ gpg --no-autostart --verify foo.txt.asc + +# install git and clone the main project on the build machine +$ sudo apt-get install -y git +$ git clone https://github.com/apache/hbase.git +# finally set up an output folder and launch a dry run. $ mkdir ~/build -$ ./dev-resources/create-release/do-release-docker.sh -d ~/build -# etc. +$ cd hbase +$ ./dev-support/create-release/do-release-docker.sh -d ~/build + +# for building the main repo specifically you can save an extra download by pointing the build +# to the local clone you just made +$ ./dev-support/create-release/do-release-docker.sh -d ~/build -r .git Review comment: nice! should this be the default ? ########## File path: dev-support/create-release/release-util.sh ########## @@ -436,6 +436,7 @@ function git_clone_overwrite { } # Writes report into cwd! +# TODO should have option for maintenance release that include LimitedPrivate in report Review comment: 👍 file an issue? ########## File path: dev-support/create-release/README.txt ########## @@ -37,15 +47,53 @@ $ sudo add-apt-repository -y \ stable" $ sudo apt-get update $ sudo apt-get install -y docker-ce docker-ce-cli containerd.io -$ sudo usermod -a -G docker $USERID +# Follow the post installation steps: https://docs.docker.com/engine/install/linux-postinstall/ +$ sudo usermod -aG docker $USER # LOGOUT and then LOGIN again so $USERID shows as part of docker group -# Copy up private key for $USERID export from laptop and import on gce. -$ gpg --import stack.duboce.net.asc -$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798 -$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400) -$ export PROJECT="${PROJECT:-hbase}" -$ git clone https://github.com/apache/${PROJECT}.git -$ cd "${PROJECT}" +# Test here by running docker's hello world as your build user +$ docker run hello-world + +# Follow the GPG guide for forwarding your gpg-agent from your local machine to the VM +# https://wiki.gnupg.org/AgentForwarding +# On the VM find out the location of the gpg agent socket and extra socket +$ gpgconf --list-dir agent-socket +/run/user/1000/gnupg/S.gpg-agent +$ gpgconf --list-dir agent-extra-socket +/run/user/1000/gnupg/S.gpg-agent.extra +# On the VM configure sshd to remove stale sockets +$ sudo bash -c 'echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config' +$ sudo systemctl restart ssh +# logout of the VM + +# Do these steps on your local machine. +# Export your public key and copy it to the VM. +# Assuming 'example.gce.host' maps to your VM's external IP (or use the IP) +$ gpg --export [email protected] > ~/gpg.example.apache.pub Review comment: really helpful docs here, thank you. ########## File path: dev-support/create-release/do-release-docker.sh ########## @@ -162,15 +219,15 @@ GIT_NAME=$GIT_NAME GIT_EMAIL=$GIT_EMAIL GPG_KEY=$GPG_KEY ASF_PASSWORD=$ASF_PASSWORD -GPG_PASSPHRASE=$GPG_PASSPHRASE RELEASE_STEP=$RELEASE_STEP API_DIFF_TAG=$API_DIFF_TAG +HOST_OS=$HOST_OS EOF -JAVA_VOL=() +JAVA_MOUNT=() if [ -n "$JAVA" ]; then echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE" - JAVA_VOL=(--volume "$JAVA:/opt/hbase-java") + JAVA_MOUNT=(--mount "type=bind,src=${JAVA},dst=/opt/hbase-java,readonly") Review comment: do we actually need java in the host environment? why not install it in the docker image? How is it the mounted java version is compatible with the docker environment? If you're running with Mac as the host and the docker image is linux... ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
