TheNeuralBit commented on a change in pull request #13308: URL: https://github.com/apache/beam/pull/13308#discussion_r532898302
########## File path: start-build-env.sh ########## @@ -0,0 +1,131 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e # exit on error + +cd "$(dirname "$0")" # connect to root + +DOCKER_DIR=dev-support/docker +DOCKER_FILE="${DOCKER_DIR}/Dockerfile" + +CONTAINER_NAME=beam-dev-${USER}-$$ + +#CPU_ARCH=$(echo "$MACHTYPE" | cut -d- -f1) +#if [ "$CPU_ARCH" = "aarch64" ]; then +# DOCKER_FILE="${DOCKER_DIR}/Dockerfile_aarch64" +#fi Review comment: can this be removed? ########## File path: dev-support/docker/beam_env_checks.sh ########## @@ -0,0 +1,120 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# SHELLDOC-IGNORE + +# ------------------------------------------------------- +function showWelcome { +# http://patorjk.com/software/taag/#p=display&f=Doom&t=Beam%20Build%20Env. +cat << "Welcome-message" + +______ ______ _ _ _ _____ +| ___ \ | ___ \ (_) | | | | ___| +| |_/ / ___ __ _ _ __ ___ | |_/ /_ _ _| | __| | | |__ _ ____ __ +| ___ \/ _ \/ _` | '_ ` _ \ | ___ \ | | | | |/ _` | | __| '_ \ \ / / +| |_/ / __/ (_| | | | | | | | |_/ / |_| | | | (_| | | |__| | | \ V / +\____/ \___|\__,_|_| |_| |_| \____/ \__,_|_|_|\__,_| \____/_| |_|\_(_) + +This is the standard Beam Developer build environment. +This has all the right tools installed required to build +Apache Beam from source. + +Welcome-message +} + +# ------------------------------------------------------- + +function showAbort { + cat << "Abort-message" + + ___ _ _ _ + / _ \| | | | (_) +/ /_\ \ |__ ___ _ __| |_ _ _ __ __ _ +| _ | '_ \ / _ \| '__| __| | '_ \ / _\` | +| | | | |_) | (_) | | | |_| | | | | (_| | +\_| |_/_.__/ \___/|_| \__|_|_| |_|\__, | + __/ | + |___/ + +Abort-message +} + +# ------------------------------------------------------- + +function failIfUserIsRoot { + if [ "$(id -u)" -eq "0" ]; # If you are root then something went wrong. + then + cat <<End-of-message + +Apparently you are inside this docker container as the user root. +Putting it simply: + + This should not occur. + +Known possible causes of this are: +1) Running this script as the root user ( Just don't ) Review comment: Let's make this more helpful - maybe it could point people to a page like this: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user ########## File path: dev-support/docker/beam_env_checks.sh ########## @@ -0,0 +1,120 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# SHELLDOC-IGNORE + +# ------------------------------------------------------- +function showWelcome { +# http://patorjk.com/software/taag/#p=display&f=Doom&t=Beam%20Build%20Env. +cat << "Welcome-message" + +______ ______ _ _ _ _____ +| ___ \ | ___ \ (_) | | | | ___| +| |_/ / ___ __ _ _ __ ___ | |_/ /_ _ _| | __| | | |__ _ ____ __ +| ___ \/ _ \/ _` | '_ ` _ \ | ___ \ | | | | |/ _` | | __| '_ \ \ / / +| |_/ / __/ (_| | | | | | | | |_/ / |_| | | | (_| | | |__| | | \ V / +\____/ \___|\__,_|_| |_| |_| \____/ \__,_|_|_|\__,_| \____/_| |_|\_(_) + +This is the standard Beam Developer build environment. +This has all the right tools installed required to build +Apache Beam from source. + +Welcome-message +} + +# ------------------------------------------------------- + +function showAbort { + cat << "Abort-message" + + ___ _ _ _ + / _ \| | | | (_) +/ /_\ \ |__ ___ _ __| |_ _ _ __ __ _ +| _ | '_ \ / _ \| '__| __| | '_ \ / _\` | +| | | | |_) | (_) | | | |_| | | | | (_| | +\_| |_/_.__/ \___/|_| \__|_|_| |_|\__, | + __/ | + |___/ Review comment: nit: The ASCII art is nice for the welcome message but I'd drop it for this and the low memory message ########## File path: start-build-env.sh ########## @@ -0,0 +1,131 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e # exit on error + +cd "$(dirname "$0")" # connect to root + +DOCKER_DIR=dev-support/docker +DOCKER_FILE="${DOCKER_DIR}/Dockerfile" + +CONTAINER_NAME=beam-dev-${USER}-$$ + +#CPU_ARCH=$(echo "$MACHTYPE" | cut -d- -f1) +#if [ "$CPU_ARCH" = "aarch64" ]; then +# DOCKER_FILE="${DOCKER_DIR}/Dockerfile_aarch64" +#fi + +docker build -t beam-build -f $DOCKER_FILE $DOCKER_DIR + +USER_NAME=${SUDO_USER:=$USER} +USER_ID=$(id -u "${USER_NAME}") + +if [ "$(uname -s)" = "Darwin" ]; then + GROUP_ID=100 +fi + +if [ "$(uname -s)" = "Linux" ]; then + GROUP_ID=$(id -g "${USER_NAME}") + # man docker-run + # When using SELinux, mounted directories may not be accessible + # to the container. To work around this, with Docker prior to 1.7 + # one needs to run the "chcon -Rt svirt_sandbox_file_t" command on + # the directories. With Docker 1.7 and later the z mount option + # does this automatically. + if command -v selinuxenabled >/dev/null && selinuxenabled; then + DCKR_VER=$(docker -v| + awk '$1 == "Docker" && $2 == "version" {split($3,ver,".");print ver[1]"."ver[2]}') + DCKR_MAJ=${DCKR_VER%.*} + DCKR_MIN=${DCKR_VER#*.} + if [ "${DCKR_MAJ}" -eq 1 ] && [ "${DCKR_MIN}" -ge 7 ] || + [ "${DCKR_MAJ}" -gt 1 ]; then + V_OPTS=:z + else + for d in "${PWD}" "${HOME}/.m2"; do + ctx=$(stat --printf='%C' "$d"|cut -d':' -f3) + if [ "$ctx" != svirt_sandbox_file_t ] && [ "$ctx" != container_file_t ]; then + printf 'INFO: SELinux is enabled.\n' + printf '\tMounted %s may not be accessible to the container.\n' "$d" + printf 'INFO: If so, on the host, run the following command:\n' + printf '\t# chcon -Rt svirt_sandbox_file_t %s\n' "$d" + fi + done + fi + fi +fi + +# Set the home directory in the Docker container. +DOCKER_HOME_DIR=${DOCKER_HOME_DIR:-/home/${USER_NAME}} + +DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3) + +docker build -t "beam-build-${USER_ID}" - <<UserSpecificDocker +FROM beam-build +RUN rm -f /var/log/faillog /var/log/lastlog +RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} +RUN groupmod -g ${DOCKER_GROUP_ID} docker +RUN useradd -g ${GROUP_ID} -G docker -u ${USER_ID} -k /root -m ${USER_NAME} -d "${DOCKER_HOME_DIR}" +RUN echo "${USER_NAME} ALL=NOPASSWD: ALL" > "/etc/sudoers.d/beam-build-${USER_ID}" +ENV HOME "${DOCKER_HOME_DIR}" +ENV GOPATH ${HOME}/beam/sdks/go/examples/.gogradle/project_gopath + +# This next command still runs as root causing the ~/.cache/go-build to be owned by root +RUN go get github.com/linkedin/goavro +RUN chown -R ${USER_NAME}:${GROUP_ID} ${HOME}/.cache +UserSpecificDocker + +echo "" +echo "Docker image build completed." +echo "==============================================================================================" +echo "" + +# If this env variable is empty, docker will be started +# in non interactive mode +DOCKER_INTERACTIVE_RUN=${DOCKER_INTERACTIVE_RUN-"-i -t"} + +DOCKER_SOCKET_MOUNT="" +if [ -S /var/run/docker.sock ]; +then + DOCKER_SOCKET_MOUNT="-v /var/run/docker.sock:/var/run/docker.sock${V_OPTS:-}" + echo "Enabling Docker support with the docker build environment." +else + echo "There is NO Docker support with the docker build environment." +fi + +COMMAND=( "$@" ) +if [ $# -eq 0 ]; +then + COMMAND=( "bash" ) +fi + +[ -d "${HOME}/.beam_docker_build_env/.gradle" ] || mkdir -p "${HOME}/.beam_docker_build_env/.gradle" + +# By mapping the .m2 directory you can do an mvn install from +# within the container and use the result on your normal +# system. And this also is a significant speedup in subsequent +# builds because the dependencies are downloaded only once. +docker run --rm=true ${DOCKER_INTERACTIVE_RUN} \ Review comment: Could you add a check that prints a helpful message if `docker` does not exist? ########## File path: dev-support/docker/beam_env_checks.sh ########## @@ -0,0 +1,120 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# SHELLDOC-IGNORE + +# ------------------------------------------------------- +function showWelcome { +# http://patorjk.com/software/taag/#p=display&f=Doom&t=Beam%20Build%20Env. +cat << "Welcome-message" + +______ ______ _ _ _ _____ +| ___ \ | ___ \ (_) | | | | ___| +| |_/ / ___ __ _ _ __ ___ | |_/ /_ _ _| | __| | | |__ _ ____ __ +| ___ \/ _ \/ _` | '_ ` _ \ | ___ \ | | | | |/ _` | | __| '_ \ \ / / +| |_/ / __/ (_| | | | | | | | |_/ / |_| | | | (_| | | |__| | | \ V / +\____/ \___|\__,_|_| |_| |_| \____/ \__,_|_|_|\__,_| \____/_| |_|\_(_) Review comment: I wonder if we could generate an ASCII firefly to put here :) https://beam.apache.org/community/mascot/ ########## File path: dev-support/docker/Dockerfile ########## @@ -0,0 +1,110 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Dockerfile for installing the necessary dependencies for building Hadoop. +# See BUILDING.txt. + +FROM ubuntu:20.04 + +ARG DEBIAN_FRONTEND=noninteractive + +WORKDIR /root + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +##### +# Disable suggests/recommends +##### +RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras +RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras + +ENV DEBIAN_FRONTEND noninteractive +ENV DEBCONF_TERSE true + + +# hadolint ignore=DL3008 +### +# Update and install common packages +### +RUN apt -q update \ + && apt install -y software-properties-common apt-utils apt-transport-https ca-certificates \ + && add-apt-repository -y ppa:deadsnakes/ppa \ + && apt-get -q install -y --no-install-recommends \ + bash-completion \ + build-essential \ + bzip2 \ + curl \ + docker.io \ + g++ \ + gcc \ + git \ + gnupg-agent \ + rsync \ + sudo \ + vim \ + locales \ + wget \ + time \ + openjdk-8-jdk \ + python3-setuptools \ + python3-pip \ + python3.5 \ + python3.6 \ + python3.7 \ + python2.7 \ Review comment: ```suggestion python3.6 \ python3.7 \ python3.8 \ ``` We no longer support python 2.7 or 3.5, and we do support 3.8 (looks like this is actually still incorrect in https://beam.apache.org/contribute/#prerequisites, I'll send a PR to fix) ########## File path: dev-support/docker/Dockerfile ########## @@ -0,0 +1,110 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Dockerfile for installing the necessary dependencies for building Hadoop. +# See BUILDING.txt. + +FROM ubuntu:20.04 + +ARG DEBIAN_FRONTEND=noninteractive + +WORKDIR /root + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +##### +# Disable suggests/recommends +##### +RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras +RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras + +ENV DEBIAN_FRONTEND noninteractive +ENV DEBCONF_TERSE true + + +# hadolint ignore=DL3008 +### +# Update and install common packages +### +RUN apt -q update \ + && apt install -y software-properties-common apt-utils apt-transport-https ca-certificates \ + && add-apt-repository -y ppa:deadsnakes/ppa \ + && apt-get -q install -y --no-install-recommends \ + bash-completion \ + build-essential \ + bzip2 \ + curl \ + docker.io \ + g++ \ + gcc \ + git \ + gnupg-agent \ + rsync \ + sudo \ + vim \ + locales \ + wget \ + time \ + openjdk-8-jdk \ + python3-setuptools \ + python3-pip \ + python3.5 \ + python3.6 \ + python3.7 \ + python2.7 \ + virtualenv \ + tox + +### +# Set the locale ( see https://stackoverflow.com/a/28406007/114196 ) +### Review comment: ```suggestion ### # Set the locale ( see https://stackoverflow.com/a/28406007/114196 ) ### # TODO(BEAM-11327): Remove the need to run tests with UTF-8 encoding ``` ########## File path: dev-support/docker/Dockerfile ########## @@ -0,0 +1,110 @@ + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Dockerfile for installing the necessary dependencies for building Hadoop. +# See BUILDING.txt. + +FROM ubuntu:20.04 + +ARG DEBIAN_FRONTEND=noninteractive + +WORKDIR /root + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +##### +# Disable suggests/recommends +##### +RUN echo APT::Install-Recommends "0"\; > /etc/apt/apt.conf.d/10disableextras +RUN echo APT::Install-Suggests "0"\; >> /etc/apt/apt.conf.d/10disableextras + +ENV DEBIAN_FRONTEND noninteractive +ENV DEBCONF_TERSE true + + +# hadolint ignore=DL3008 +### +# Update and install common packages +### +RUN apt -q update \ + && apt install -y software-properties-common apt-utils apt-transport-https ca-certificates \ + && add-apt-repository -y ppa:deadsnakes/ppa \ + && apt-get -q install -y --no-install-recommends \ + bash-completion \ + build-essential \ + bzip2 \ + curl \ + docker.io \ + g++ \ + gcc \ + git \ + gnupg-agent \ + rsync \ + sudo \ + vim \ + locales \ + wget \ + time \ + openjdk-8-jdk \ + python3-setuptools \ + python3-pip \ + python3.5 \ + python3.6 \ + python3.7 \ + python2.7 \ + virtualenv \ + tox + +### +# Set the locale ( see https://stackoverflow.com/a/28406007/114196 ) +### +RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \ + locale-gen +ENV LANG en_US.UTF-8 +ENV LANGUAGE en_US:en +ENV LC_ALL en_US.UTF-8 + +### +# Set Python3.6 as default +### +RUN alias python=python3.6 + +### +# Install grpcio-tools mypy-protobuf for `python3 sdks/python/setup.py sdist` to work +### +RUN pip3 install grpcio-tools mypy-protobuf Review comment: What happens without these? If they are necessary for a build environment we should add them to https://beam.apache.org/contribute/#prerequisites as well ########## File path: CHANGES.md ########## @@ -97,6 +97,7 @@ * Added support for avro payload format in Beam SQL Pubsub Table ([BEAM-5504](https://issues.apache.org/jira/browse/BEAM-5504)) * Added option to disable unnecessary copying between operators in Flink Runner (Java) ([BEAM-11146](https://issues.apache.org/jira/browse/BEAM-11146)) * Added CombineFn.setup and CombineFn.teardown to Python SDK. These methods let you initialize the CombineFn's state before any of the other methods of the CombineFn is executed and clean that state up later on. If you are using Dataflow, you need to enable Dataflow Runner V2 by passing `--experiments=use_runner_v2` before using this feature. ([BEAM-3736](https://issues.apache.org/jira/browse/BEAM-3736)) +* Added Docker based developer build environment. ([BEAM-10891](https://issues.apache.org/jira/browse/BEAM-10891)) Review comment: I don't think this belongs in CHANGES.md, it is intended to communicate changes in the Beam release to users, but this change is just for developers. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
