Thanks Alex, I'll take a look On Mon, Dec 14, 2020 at 12:48 PM Alex Kosolapov <alex.kosola...@akvelon.com> wrote:
> Hi! > > > > Wanted to send a note that PR with changes submitted > https://github.com/apache/beam/pull/13545 and ready for code review > > > > Thanks, > > Alex > > *From: *Alex Kosolapov <alex.kosola...@akvelon.com> > *Date: *Thursday, December 10, 2020 at 3:44 PM > *To: *"dev@beam.apache.org" <dev@beam.apache.org> > *Subject: *Re: Docker Development Environment > > > > Hi Brian, > > > > Yes - will update the script, created JIRA to track > https://issues.apache.org/jira/browse/BEAM-11438 > > > > I was thinking about dscl initially too - and dscl will require user to > create docker group in this case as you noted. For some users that might be > an extra manual step to do (group not created automatically by Docker > install). > > > > Can approaches be combined? > > 1. If docker group created – use docker group’s gid > 2. If docker group not created – assign some default, e.g. 1000 or > some other safe way of handling “no docker group” case > > > > Thanks, > > Alex > > > > *From: *Brian Hulette <bhule...@google.com> > *Reply-To: *"dev@beam.apache.org" <dev@beam.apache.org> > *Date: *Wednesday, December 9, 2020 at 12:21 PM > *To: *dev <dev@beam.apache.org> > *Subject: *Re: Docker Development Environment > > > > Hey Alex, > Thanks for trying this out on macOS! Some responses inline > > > > On Tue, Dec 8, 2020 at 11:42 AM Alex Kosolapov <alex.kosola...@akvelon.com> > wrote: > > Hi! > > > > Thank you for creating Docker build environment - makes build environment > setup so much easier! > > > > I ran start-build-env.sh on a macOS, and I ran into some items that wanted > to share + propose how to improve Docker build environment for macOS > support: > > 1. ./start-build-env.sh: line 75: getent: command not found and script > build error downstream. > > > > Step 26/26 : RUN echo '. /scripts/beam_env_checks.sh' >> > /root/.bash_aliases > > ---> Using cache > > ---> fe48b8b26e91 > > Successfully built fe48b8b26e91 > > Successfully tagged beam-build:latest > > *./start-build-env.sh: line 75: getent: command not found* > > Sending build context to Docker daemon 2.56kB > > Step 1/10 : FROM beam-build > > ---> fe48b8b26e91 > > Step 2/10 : RUN rm -f /var/log/faillog /var/log/lastlog > > ---> Using cache > > ---> 11ea31f2099e > > Step 3/10 : RUN groupadd --non-unique -g 100 alex > > ---> Using cache > > ---> cf3f56f51d9f > > *Step 4/10 : RUN groupmod -g docker* > > * ---> Running in 5d5bc473be3b* > > *groupmod: invalid group ID 'docker'* > > *The command '/bin/bash -o pipefail -c groupmod -g docker' returned a > non-zero code: 3* > > > > I investigated the issues and found that start-build-env.sh expected group > "docker" created in the system groups. Creating a docker group is an > optional post install step ( > https://docs.docker.com/engine/install/linux-postinstall/), another relevant > reference to forum <https://forums.docker.com/t/no-more-docker-group/9123> > gives a hint of using staff group on macOS. Solution that worked for me was > pick another GID for DOCKER_GROUP_ID on Mac, e.g. > > #DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3) > > > > if [ "$(uname -s)" = "Linux" ]; then > > DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3) > > fi > > if [ "$(uname -s)" = "Darwin" ]; then > > DOCKER_GROUP_ID=1000 > > fi > > > > I think everyone using this container should do that postinstall step so > that docker doesn't have to be run as root. We actually point to it in an > error message if we detect the script is run as root. I think the real > issue here is that getent doesn't exist on MacOS, it looks like there is an > alternative approach though: > https://stackoverflow.com/questions/10910096/what-is-the-command-to-get-groupid-of-a-group-name-in-mac-or-linux > > Would you be ok updating the script to use that approach on macOS? This > would still require users to make a docker group. > > > > > > 1. Step 10/10 : RUN chown -R alex: /Users/alex/.cache > > ---> Running in f09e2bb0e045 > > chown: cannot access '/Users/alex/.cache': No such file or directory > > The command '/bin/bash -o pipefail -c chown -R alex: /Users/alex/.cache' > returned a non-zero code: 1 > > > > Fix for this was to change from HOME to DOCKER_HOME_DIR: > > RUN chown -R ${USER_NAME}:${GROUP_ID} > ${DOCKER_HOME_DIR}/.cache > > ENV GOPATH > ${DOCKER_HOME_DIR}/beam/sdks/go/examples/.gogradle/project_gopath > > Ah yes this definitely looks like a bug. It just happened to work when > running from a machine where the HOME dir was the same as in the container. > This one definitely makes sense. > > > > If these proposed solutions make sense I will create JIRA ticket and > submit these improvements for Docker build support on macOS. > > > > Thank you, > > Alex Kosolapov > > > > *From: *Brian Hulette <bhule...@google.com> > *Reply-To: *"dev@beam.apache.org" <dev@beam.apache.org> > *Date: *Friday, December 4, 2020 at 1:15 PM > *To: *dev <dev@beam.apache.org> > *Cc: *Omar Ismail <omarism...@google.com> > *Subject: *Re: Docker Development Environment > > > > I think https://github.com/apache/beam/pull/13308 is about ready to > merge. One question was whether or not to install pyenv in the container - > I think we should try to do without it. Users of this environment will > already be operating within a container, so they shouldn't need pyenv to > create isolated python environments, they could just use the container's > system python. > > > > The two issues Alex mentioned are still outstanding, but they seem to be > issues with ./gradlew check unrelated to the container. I filed BEAM-11402 > [1] to track this separately. > > > > [1] https://issues.apache.org/jira/browse/BEAM-11402 > > > > On Mon, Nov 30, 2020 at 11:43 AM Alex Amato <ajam...@google.com> wrote: > > If any of these are suitable for at least some development. I propose we > merge them, and update them with fixes later. Rather than trying to get > things 100% working in the first PR. > > > > Looks like this one was opened in early Sept, and never got merged. Which > is a pretty long time. Perhaps abandoned for the later? > > https://github.com/apache/beam/pull/12837 > > > > This one looks like its just failing on just a few tests (Which may be > addressed soon, but the PR was opened 19 days ago). > > https://github.com/apache/beam/pull/13308 > > (Can we set a deadline for this? And just say merge it by the end of the > week, regardless if the last two tests can be fixed or not?) > > > > (Would like to nudge this along, as it's been a pain point for many for a > while now). > > > > Thanks for the work here Niels, Omar and Sam. > > Looking forward to giving this a try :) > > > > > > On Mon, Nov 30, 2020 at 11:32 AM Brian Hulette <bhule...@google.com> > wrote: > > I agree this is a good idea. I remember my first experience with Beam > development - I ran through the steps at [1] and had `./gradlew check` > fail. I don't think I ever got it working before moving on and just running > more specific tasks. > > It would be great if we had a reliable way for new contributors to > establish an environment that can successfully run `./gradlew check`. > > > > Niels Basjes' PR (https://github.com/apache/beam/pull/13308) seems to be > close to that, so I think we should focus on getting that working and > iterate from there. Omar concurred with that in > https://github.com/apache/beam/pull/12837. > > > [1] https://beam.apache.org/contribute/#development-setup > > > > > > On Wed, Nov 25, 2020 at 3:39 PM Ahmet Altay <al...@google.com> wrote: > > Thank you for doing this. > > > > I have seen a few related PRs. Connecting them here in case these efforts > could be combined: > > - https://github.com/apache/beam/pull/12837 (/cc +Omar Ismail > <omarism...@google.com> ) > > - https://github.com/apache/beam/pull/13308 > > > > Ahmet > > > > On Wed, Nov 25, 2020 at 2:53 PM Sam Rohde <sro...@google.com> wrote: > > Hi All, > > > > I got tired of my local dev environment being ruined by updates so I made > a container for Apache Beam development work. What this does is create a > Docker container from the Ubuntu Groovy image and load it up with all the > necessary libraries/utilities for Apache Beam development. Then I run an > interactive shell in the Docker container where I do my work. > > > > This is a nice way for new contributors to easily get started. However > with the container in its current form, I don't know if this will help > other people because it is tied closely with my workflow (using VIM, > YouCompleteMe, for Python). But I think it can be a nice starting point for > improvements. For example: > > - Sharing the host display with Docker to start GUI applications (like > IntelliJ) in the container > - Adding Golang development support > > Here's a draft PR <https://github.com/apache/beam/pull/13430>, let me > know what you think, how it can be improved, and whether it's a good idea > for us to have a dev container like this. > > > > Regards, > > Sam > > > >
