[
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=127162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127162
]
ASF GitHub Bot logged work on BEAM-4838:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jul/18 08:53
Start Date: 25/Jul/18 08:53
Worklog Time Spent: 10m
Work Description: lgajowy closed pull request #5989: [BEAM-4838] Add
dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/dockerized-jenkins/Dockerfile
b/.test-infra/dockerized-jenkins/Dockerfile
new file mode 100644
index 00000000000..f307ca0be2f
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/Dockerfile
@@ -0,0 +1,40 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+FROM jenkins/jenkins:lts
+
+# Jenkins image specifies /var/jenkins_home as volume.
+# For matter of copying container, we want to avoid it.
+# We have to utilize different folder because docker doesn't
+# provide "UNVOLUME" functionality.
+ENV JENKINS_HOME=/var/jenkins_real_home/
+
+# Pre-install plugins specified in plugins.txt
+COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
+RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
+
+# Copy default configuration and sample seed job.
+COPY --chown=jenkins basic-security.groovy
/usr/share/jenkins/ref/init.groovy.d/
+COPY --chown=jenkins
javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
/var/jenkins_real_home/
+COPY --chown=jenkins config.xml /var/jenkins_real_home/
+COPY --chown=jenkins seedjobconfig.xml
/var/jenkins_real_home/jobs/sample_seed_job/config.xml
+
+# Disable setup wizard
+RUN echo $JENKINS_VERSION >
/usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state
+RUN echo $JENKINS_VERSION >
/usr/share/jenkins/ref/jenkins.install.InstallUtil.lastExecVersion
+
diff --git a/.test-infra/dockerized-jenkins/README.md
b/.test-infra/dockerized-jenkins/README.md
new file mode 100644
index 00000000000..5927c6f2421
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/README.md
@@ -0,0 +1,168 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Local Jenkins Setup
+
+Setting up a local Jenkins instance is useful for testing Jenkins job changes
+without affecting the production instance. Using Docker makes the setup very
+straightforward.
+
+As of now, this setup allows you to test Job DSL scripts. Imported Beam jobs
+will not succeed due to some tools (like gcloud) and credentials missing.
+
+## Jenkins on Docker
+
+### Requirements
+If you haven't yet, [install Docker on your
machine.](https://docs.docker.com/install/)
+
+### Setup using provided scripts
+
+**WARNING: Not for production use.**
+
+*INFO: Changing admin credentials is advised.*
+
+You can utilize scripts in this folder to build Docker container with Jenkins,
+pre-installed plugins and some basic configuration.
+
+```bash
+docker build -t beamjenkins .
+docker run -p 127.0.0.1:8080:8080 beamjenkins:latest
+```
+* plugins.txt -- Pre-assembled list of plugins that are available on Apache
+ Jenkins. This list does not contain versions and might not have all the
+ plugins available. If you find some plugin missing, or having and incorrect
+ version, you can check
+ [infra wiki
list](https://cwiki.apache.org/confluence/display/INFRA/Jenkins+Plugin+Upgrades)
+ for plugin updates history. Feel free to update this list with added/removed
+ plugins.
+* docker build -- builds image with name beamjenkins based on Dockerfile
located
+ in current folder.
+* docker run -- creates and starts new container.
+ * The `-p 127.0.0.1:8080:8080` parameter to `docker run` ensures that your
+ Jenkins instance will only be available via localhost and not your
machine
+ hostname.
+* Default credentials: admin:jenadmin (Suggested to change these in
Dockerfile.)
+
+Image built via this scripts will contain all required plugins, basic
+configuration listed in manual setup instructions below and also remap
+JENKINS_HOME, so it does not map to a volume, but is part of container instead.
+
+This last action will allow you to persist changes you do to your container:
+
+```bash
+docker ps -a
+docker commit <container_id> <new_image_name>
+docker run -p 127.0.0.1:8080:8080 <new_image_name>
+```
+* docker ps -- list containers.
+* docker commit -- creates new image based on container.
+
+Such approach may be handy for debugging, but it is highly advised to keep all
+data in docker volumes.
+
+## Manual setup
+
+When invoking Docker, you can map a local directory to persist state and keep
+your Jenkins configuration even after machine reboot.
+
+```bash
+JENKINS_HOME=$(readlink --canonicalize-missing ~/jenkins/homedir)
+
+mkdir -p ${JENKINS_HOME}
+chmod -R 777 ${JENKINS_HOME}
+docker run -p 127.0.0.1:8080:8080 -v ${JENKINS_HOME}:/var/jenkins_home
jenkins/jenkins:lts
+```
+
+Running this command will bring up your Jenkins instance at
+http://127.0.0.1:8080. And map jenkins_home of running container to
JENKINS_HOME
+on your OS file system.
+
+You can setup your Jenkins instance to look like the Apache Beam Jenkins using
+[these steps](#beam-config). (Running Beam Job DSL groovy scripts)
+
+### Forking your Jenkins instance
+
+Later you can fork a new Docker container by copying the contents of the mapped
+directory and starting Jenkins from the new copy.
+
+This might be useful in case if you want to test new plugins/configuration and
+want to have a clean roll-back option.
+
+```bash
+JENKINS_NEWHOME=$(readlink --canonicalize-missing ~/jenkins/homedir_v2)
+
+mkdir -p ${JENKINS_NEWHOME}
+cp -R ${JENKINS_HOME} ${JENKINS_NEWHOME}
+JENKINS_HOME=${JENKINS_NEWHOME}
+
+chmod -R 777 ${JENKINS_HOME}
+docker run -p 127.0.0.1:8080:8080 -v ${JENKINS_HOME}:/var/jenkins_home
jenkins/jenkins:lts
+```
+* The `-p 127.0.0.1:8080:8080` parameter to `docker run` ensures that your
+ Jenkins instance will only be available via localhost and not your
machine
+ hostname.
+
+### Running Beam Job DSL groovy scripts {#beam-config}
+
+On Beam we configure Jenkins jobs via groovy job dsl scripts. If you want to
run
+those on your docker instance of Jenkins, you will need to do some setup on top
+of installing default plugins:
+
+1. Install following plugins
+ 1. Environment Injector
+ 1. GitHub pull request build (ghprb)
+ 1. Groovy
+ 1. Job DSL
+ 1. Node and Label parameter
+ 1. (Optional) CustomHistory: This will allow you to easily generate job
+ diffs as you make changes.
+1. Add "beam" label to Jenkins instance
+ 1. Go to Manage Jenkins -> Configure System
+ 1. Type "beam" under "Labels" field.
+1. Disable script security. This way you will not have to approve all the
+ scripts.
+ 1. Go to Manage Jenkins -> Configure Global Security
+ 1. Unmark "Enable script security for Job DSL scripts"
+1. Set up Job DSL jobs import job. (Seed job)
+ 1. Refer to seedjobconfig.xml for parameters.
+ 1. Go to Jenkins -> New Item -> Freestyle project
+ 1. Build step: Process Job DSLs
+
+## Additional docker hints
+
+### Running image vs starting container
+
+When you execute `docker run`, it creates a new container for your image.
+Instead of creating a new container every time, you can restart a
+previously-exited container:
+
+```bash
+docker ps # Look for the previous container_id
+docker start <container_id>
+```
+
+One of the benefits of restarting is that it will have all the data and
+configuration parameters you have assigned to it upon creation.
+
+### Delete all images and containers
+```bash
+docker rm $(docker ps -a -q)
+docker rmi $(docker images -q)
+docker volume prune
+```
diff --git a/.test-infra/dockerized-jenkins/basic-security.groovy
b/.test-infra/dockerized-jenkins/basic-security.groovy
new file mode 100644
index 00000000000..669938256ce
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/basic-security.groovy
@@ -0,0 +1,24 @@
+#!groovy
+
+import jenkins.model.*
+import hudson.security.*
+import jenkins.install.InstallState
+
+def instance = Jenkins.getInstance()
+
+// Create user with custom pass
+def hudsonRealm = new HudsonPrivateSecurityRealm(false)
+hudsonRealm.createAccount('admin','jenadmin')
+instance.setSecurityRealm(hudsonRealm)
+
+def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
+strategy.setAllowAnonymousRead(false)
+instance.setAuthorizationStrategy(strategy)
+
+// Disable setup wizard
+if (!instance.installState.isSetupComplete()) {
+ InstallState.INITIAL_SETUP_COMPLETED.initializeState()
+}
+
+instance.save()
+
diff --git a/.test-infra/dockerized-jenkins/config.xml
b/.test-infra/dockerized-jenkins/config.xml
new file mode 100644
index 00000000000..cac2041db3a
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/config.xml
@@ -0,0 +1,41 @@
+<?xml version='1.1' encoding='UTF-8'?>
+<hudson>
+ <disabledAdministrativeMonitors/>
+ <version>2.121.1</version>
+ <installStateName>DOWNGRADE</installStateName>
+ <numExecutors>2</numExecutors>
+ <mode>NORMAL</mode>
+ <useSecurity>true</useSecurity>
+ <authorizationStrategy
class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
+ <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
+ </authorizationStrategy>
+ <securityRealm class="hudson.security.HudsonPrivateSecurityRealm">
+ <disableSignup>true</disableSignup>
+ <enableCaptcha>false</enableCaptcha>
+ </securityRealm>
+ <disableRememberMe>false</disableRememberMe>
+ <projectNamingStrategy
class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
+ <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>
+ <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
+ <markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
+ <jdks/>
+ <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
+ <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
+ <clouds/>
+ <quietPeriod>5</quietPeriod>
+ <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
+ <views>
+ <hudson.model.AllView>
+ <owner class="hudson" reference="../../.."/>
+ <name>all</name>
+ <filterExecutors>false</filterExecutors>
+ <filterQueue>false</filterQueue>
+ <properties class="hudson.model.View$PropertyList"/>
+ </hudson.model.AllView>
+ </views>
+ <primaryView>all</primaryView>
+ <slaveAgentPort>50000</slaveAgentPort>
+ <label>beam</label>
+ <nodeProperties/>
+ <globalNodeProperties/>
+</hudson>
\ No newline at end of file
diff --git
a/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
b/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
new file mode 100644
index 00000000000..6d808a28b49
--- /dev/null
+++
b/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
@@ -0,0 +1,5 @@
+<?xml version='1.1' encoding='UTF-8'?>
+<javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration
plugin="[email protected]">
+ <category class="jenkins.model.GlobalConfigurationCategory$Security"/>
+ <useScriptSecurity>false</useScriptSecurity>
+</javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration>
\ No newline at end of file
diff --git a/.test-infra/dockerized-jenkins/plugins.txt
b/.test-infra/dockerized-jenkins/plugins.txt
new file mode 100644
index 00000000000..d61bb404bb9
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/plugins.txt
@@ -0,0 +1,112 @@
+CustomHistory
+gradle
+nodelabelparameter
+envinject
+build-timeout
+throttle-concurrents
+
+active-directory
+analysis-core
+apache-httpcomponents-client-4-api
+aws-java-sdk
+blueocean
+blueocean-autofavorite
+blueocean-bitbucket-pipeline
+blueocean-commons
+blueocean-config
+blueocean-core-js
+blueocean-dashboard
+blueocean-events
+blueocean-git-pipeline
+blueocean-github-pipeline
+blueocean-i18n
+blueocean-jira
+blueocean-jwt
+blueocean-personalization
+blueocean-pipeline-api-impl
+blueocean-pipeline-editor
+blueocean-pipeline-scm-api
+blueocean-rest
+blueocean-rest-impl
+blueocean-web
+bouncycastle-api
+branch-api
+build-monitor-plugin
+build-name-setter
+build-time-blame
+cloudbees-bitbucket-branch-source
+cloudbees-folder
+clover
+cobertura
+config-file-provider
+copyartifact
+credentials
+credentials-binding
+cvs
+docker-commons
+docker-workflow
+durable-task
+email-ext
+envinject-api
+favorite
+ghprb
+git
+git-client
+github
+github-api
+github-branch-source
+github-pr-coverage-status
+htmlpublisher
+jackson2-api
+jacoco
+jenkins-design-language
+jira
+job-dsl
+ldap
+locale
+lockable-resources
+mailer
+matrix-project
+maven-plugin
+mercurial
+metrics
+monitoring
+node-iterator-api
+openid
+performance
+pipeline-graph-analysis
+pipeline-maven
+pipeline-model-api
+pipeline-model-definition
+pipeline-model-extensions
+pipeline-rest-api
+pipeline-stage-tags-metadata
+pipeline-stage-view
+pipeline-utility-steps
+plot
+promoted-builds
+rebuild
+resource-disposer
+scm-api
+script-security
+sloccount
+sonar
+ssh-credentials
+ssh-slaves
+subversion
+support-core
+testng-plugin
+timestamper
+token-macro
+translation
+urltrigger
+warnings
+workflow-api
+workflow-basic-steps
+workflow-cps
+workflow-job
+workflow-multibranch
+workflow-step-api
+workflow-support
+xunit
+
diff --git a/.test-infra/dockerized-jenkins/seedjobconfig.xml
b/.test-infra/dockerized-jenkins/seedjobconfig.xml
new file mode 100644
index 00000000000..fa7da63eb9b
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/seedjobconfig.xml
@@ -0,0 +1,48 @@
+<?xml version='1.1' encoding='UTF-8'?>
+<project>
+ <actions/>
+ <description></description>
+ <keepDependencies>false</keepDependencies>
+ <properties/>
+ <scm class="hudson.plugins.git.GitSCM" plugin="[email protected]">
+ <configVersion>2</configVersion>
+ <userRemoteConfigs>
+ <hudson.plugins.git.UserRemoteConfig>
+ <url>https://github.com/apache/beam.git</url>
+ </hudson.plugins.git.UserRemoteConfig>
+ </userRemoteConfigs>
+ <branches>
+ <hudson.plugins.git.BranchSpec>
+ <name>master</name>
+ </hudson.plugins.git.BranchSpec>
+ </branches>
+
<doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
+ <submoduleCfg class="list"/>
+ <extensions>
+ <hudson.plugins.git.extensions.impl.CleanCheckout/>
+ </extensions>
+ </scm>
+ <canRoam>true</canRoam>
+ <disabled>false</disabled>
+ <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+ <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+ <triggers/>
+ <concurrentBuild>false</concurrentBuild>
+ <builders>
+ <javaposse.jobdsl.plugin.ExecuteDslScripts plugin="[email protected]">
+ <targets>.test-infra/jenkins/job_*.groovy</targets>
+ <usingScriptText>false</usingScriptText>
+ <sandbox>false</sandbox>
+ <ignoreExisting>false</ignoreExisting>
+ <ignoreMissingFiles>false</ignoreMissingFiles>
+ <failOnMissingPlugin>false</failOnMissingPlugin>
+ <unstableOnDeprecation>false</unstableOnDeprecation>
+ <removedJobAction>DISABLE</removedJobAction>
+ <removedViewAction>IGNORE</removedViewAction>
+ <removedConfigFilesAction>IGNORE</removedConfigFilesAction>
+ <lookupStrategy>JENKINS_ROOT</lookupStrategy>
+ </javaposse.jobdsl.plugin.ExecuteDslScripts>
+ </builders>
+ <publishers/>
+ <buildWrappers/>
+</project>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 127162)
Time Spent: 5h 20m (was: 5h 10m)
> Prepare instructions for running Jenkins locally.
> -------------------------------------------------
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
> Issue Type: Improvement
> Components: build-system
> Reporter: Mikhail Gryzykhin
> Assignee: Mikhail Gryzykhin
> Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build
> server. One of the approaches available is to spin up Jenkins on your local
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other
> developers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
