[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426674#comment-16426674 ] Ismaël Mejía commented on BEAM-3954: Ok so it seems we are aligned then [~alanmyrvold]. i know docker well so I can help there, however I am a jenkins newbie, but I can try to help there also if needed. [~yifanzou] The image adds whatever you pass as a parameter at build time and the default is github's master, but you can pass a PR, a branch, etc for details take a look at the [documentation|[https://beam.apache.org/contribute/docker-images/].] How do you guys plan to tackle the problem of downloading all the deps on each build ? I did this via volume mounting but still not convinced is the best way, if no the build takes ages. We have to also pay attention at how we run [java in containers|https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/]. > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420935#comment-16420935 ] Alan Myrvold commented on BEAM-3954: Our goal is to have a reproducible build environment on jenkins that can easily be customized to provide new versions of underlying tools. If we can use this to help releases too, that'd be great. During the last release, Robert ran into issues running build commands locally/ > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420911#comment-16420911 ] yifan zou commented on BEAM-3954: - [~iemejia], I believe it will be similar to the dockerfiles you wrote. But instead of pack Beam git repo into the images, we will just wrap all frameworks and tools such as maven-3-* and google-cloud-sdk and run beam sdk within container. We will not only run the beam sdk against the HEAD, but also nightly snapshot, release candidates and dev's feature branches. I am not sure if I understand your dockerfiles correctly since I am still new to docker, so correct me if I was wrong. At this moment, I would like to write a separate dockerfile for jenkins, any thoughts [~alanmyrvold]? > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419662#comment-16419662 ] Ismaël Mejía commented on BEAM-3954: Thanks for the explanation [~yifanzou], probably worth writing the goal of the issue next time in the description of the issue (even if the title gives a really good hint). Did you see the Dockerfiles I mention ? Do you think it makes sense to unify both? This way other people can benefit also of this to have a way to use the exact same 'agents' as jenkins would do. (Those scripts were my poor man jenkins when I had issues with my OS to validate some PRs). > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419593#comment-16419593 ] yifan zou commented on BEAM-3954: - [~iemejia], We want to automate release validations, which runs some examples on Jenkins. This requires us to install some packages on beam Jenkins agents such as google cloud sdk. Since Jenkins nodes are not under our management, we are trying to figure out how to pack all stuffs we need into a docker image and make jenkins job build in the container. This will benefit us on scaling automation test to cover more examples and languages in short term, and customization on Jenkins without getting apache-infra in long term. > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized
[ https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418914#comment-16418914 ] Ismaël Mejía commented on BEAM-3954: Can you please give more details on this ticket, what is the goal ? I am curious because I worked some months ago on having reproducible build environments via docker and ended up with some Dockerfiles. It would be nice if we could have those used by jenkins (if this is the goal). For ref. [https://github.com/apache/beam/tree/master/sdks/java/build-tools/src/main/resources/docker] CC: [~alanmyrvold] in case we can also use this to make the releases more robust too. > Get Jenkins agents dockerized > -- > > Key: BEAM-3954 > URL: https://issues.apache.org/jira/browse/BEAM-3954 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
