[
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=171188&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171188
]
ASF GitHub Bot logged work on BEAM-3954:
----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Nov/18 19:51
Start Date: 30/Nov/18 19:51
Worklog Time Spent: 10m
Work Description: stale[bot] closed pull request #6254: Do Not Merge
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/dockerized-jenkins/Dockerfile
b/.test-infra/dockerized-jenkins/Dockerfile
index f307ca0be2f4..61bd83a41c09 100644
--- a/.test-infra/dockerized-jenkins/Dockerfile
+++ b/.test-infra/dockerized-jenkins/Dockerfile
@@ -18,6 +18,20 @@
FROM jenkins/jenkins:lts
+# Install docker client.
+USER root
+RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+RUN apt-key fingerprint 0EBFCD88 | grep '9DC8 5822 9FC7 DD38 854A E2D8 8D81
803C 0EBF CD88'
+RUN apt-get update && apt-get install -y \
+ software-properties-common \
+ apt-transport-https \
+ && rm -rf /var/lib/apt/lists/*
+RUN add-apt-repository "deb [arch=amd64]
https://download.docker.com/linux/debian stretch stable"
+RUN apt-get update && apt-get install -y \
+ docker-ce \
+ && rm -rf /var/lib/apt/lists/*
+USER jenkins
+
# Jenkins image specifies /var/jenkins_home as volume.
# For matter of copying container, we want to avoid it.
# We have to utilize different folder because docker doesn't
diff --git a/.test-infra/dockerized-jenkins/README.md
b/.test-infra/dockerized-jenkins/README.md
index 5927c6f24213..ce6437cc255a 100644
--- a/.test-infra/dockerized-jenkins/README.md
+++ b/.test-infra/dockerized-jenkins/README.md
@@ -166,3 +166,20 @@ docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume prune
```
+
+### Allow Jenkins to access local Docker install
+ This setup assumes you have docker installed and a `docker` group that's
allowed
+to write to its socket file:
+ ```bash
+$ ls -l /var/run/docker.sock
+srw-rw---- 1 root docker 0 Aug 14 15:42 /var/run/docker.sock
+```
+ The following additional arguments to `docker run` will make the socket file
+available to the Jenkins container, and make the `jenkins` user a member of the
+`docker` group (using its ID on the host).
+ ```bash
+docker run -p 127.0.0.1:8080:8080 \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ --group-add $(getent group docker | cut -d: -f3) \
+ beamjenkins:latest
+```
\ No newline at end of file
diff --git a/.test-infra/dockerized-jenkins/build-env/Dockerfile
b/.test-infra/dockerized-jenkins/build-env/Dockerfile
new file mode 100644
index 000000000000..50d287192d40
--- /dev/null
+++ b/.test-infra/dockerized-jenkins/build-env/Dockerfile
@@ -0,0 +1,71 @@
+###############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###############################################################################
+
+FROM openjdk:8-jdk
+MAINTAINER Beam [email protected]
+
+# Install dependencies for Jenkins Python tests
+RUN apt-get update && \
+ apt-get install -y \
+ python-pip \
+ python-virtualenv \
+ python-dev \
+ python-tox \
+ maven \
+ rsync \
+ time \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN wget
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-189.0.0-linux-x86_64.tar.gz
-O gcloud.tar.gz && \
+tar xf gcloud.tar.gz && \
+./google-cloud-sdk/install.sh --quiet
+ENV PATH="/google-cloud-sdk/bin:${PATH}"
+RUN gcloud components update --quiet || echo 'gcloud components update failed'
&& \
+ rm gcloud.tar.gz
+
+
+# Add the entrypoint script which install gcloud sdk
+# COPY docker-entrypoint.sh /usr/local/bin/
+# RUN ln -s usr/local/bin/docker-entrypoint.sh /entrypoint.sh
+# ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
+
+# Create beam user to validate the build on user space
+ENV USER=beam-jenkins \
+ UID=910\
+ GID=1002 \
+ HOME=/home/user
+RUN groupadd --system --gid=$GID $USER; \
+ useradd --system --uid=$UID --gid $USER $USER;
+RUN mkdir -p $HOME; \
+ chown -R $USER:$USER $HOME; \
+ chmod 777 $HOME;
+USER $USER
+WORKDIR $HOME
+
+#COPY my-first-project-190318.json $HOME/credentials.json
+#RUN gcloud auth activate-service-account --key-file=credentials.json && \
+# gcloud config set project my-first-project-190318 && \
+# rm credentials.json
+
+
+#ARG URL=https://github.com/apache/beam
+#
+#RUN git clone $URL beam; \
+# cd beam; \
+# git config --local --add remote.origin.fetch
'+refs/pull/*/head:refs/remotes/origin/pr/*'; \
+# git fetch --quiet --all;
diff --git a/.test-infra/jenkins/PrecommitJobBuilder.groovy
b/.test-infra/jenkins/PrecommitJobBuilder.groovy
index 49b4f9817896..eebc220cbb04 100644
--- a/.test-infra/jenkins/PrecommitJobBuilder.groovy
+++ b/.test-infra/jenkins/PrecommitJobBuilder.groovy
@@ -98,6 +98,11 @@ class PrecommitJobBuilder {
'master',
timeoutMins,
allowRemotePoll) // needed for included regions PR triggering; see
[JENKINS-23606]
+// wrappers {
+// buildInDocker {
+// dockerfile('src/.test-infra/dockerized-jenkins/build-env/',
'Dockerfile')
+// }
+// }
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
diff --git a/.test-infra/jenkins/job_Stale_Docker_Cleanup.groovy
b/.test-infra/jenkins/job_Stale_Docker_Cleanup.groovy
new file mode 100644
index 000000000000..5c04f333553f
--- /dev/null
+++ b/.test-infra/jenkins/job_Stale_Docker_Cleanup.groovy
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+
+// These jobs list details about each beam runner, to clarify what software
+// is on each machine.
+def nums = 6//1..16
+nums.each {
+ def machine = "beam${it}"
+
+ job("beam_Docker_Image_cleanup_${machine}") {
+ description('Runs Docker Image Cleanup.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Allows triggering this build against pull requests.
+ commonJobProperties.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Jenkins Docker Image Cleanup',
+ "Run Docker Image Cleanup ${machine}")
+
+ // This is a job that runs weekly.
+// commonJobProperties.setAutoJob(
+// delegate,
+// '0 12 * * 1')
+
+ parameters {
+ nodeParam('TEST_HOST') {
+ description("Select test host ${machine}")
+ defaultNodes([machine])
+ allowedNodes([machine])
+ trigger('multiSelectionDisallowed')
+ eligibility('IgnoreOfflineNodeEligibility')
+ }
+ }
+ steps {
+ shell('docker image ls && docker rmi $(docker images --filter
"dangling=true" -q --no-trunc)')
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_reproducible_env_test.groovy
b/.test-infra/jenkins/job_reproducible_env_test.groovy
new file mode 100644
index 000000000000..413f8c1f1ef0
--- /dev/null
+++ b/.test-infra/jenkins/job_reproducible_env_test.groovy
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+
+job('beam_Reproducible_Env_Test') {
+ description('Runs Beam Reproducible Environment Test.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Allows triggering this build against pull requests.
+ commonJobProperties.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Run Reproducible Env Test',
+ 'Run Reproducible Env Test')
+
+ environmentVariables {
+ env('JAVA_HOME', '/docker-java-home')
+ }
+
+ wrappers {
+ buildInDocker {
+ dockerfile('src/.test-infra/dockerized-jenkins/build-env/',
'Dockerfile')
+ }
+ colorizeOutput()
+ }
+ def machine = "beam6"
+ parameters {
+ nodeParam('TEST_HOST') {
+ description("Select test host ${machine}")
+ defaultNodes([machine])
+ allowedNodes([machine])
+ trigger('multiSelectionDisallowed')
+ eligibility('IgnoreOfflineNodeEligibility')
+ }
+ }
+ steps {
+ shell('pwd && cd src && echo $UID && echo $USER && pip --version &&
./gradlew :pythonPreCommit --info --continue')
+// shell('pwd && cd src && echo $UID && echo $USER && pip --version &&
python .test-infra/jenkins/userid_test.py')
+// gradle {
+// rootBuildScriptDir(commonJobProperties.checkoutDir)
+// tasks(':pythonPreCommit')
+// commonJobProperties.setGradleSwitches(delegate)
+// }
+ }
+
+}
diff --git a/.test-infra/jenkins/userid_test.py
b/.test-infra/jenkins/userid_test.py
new file mode 100644
index 000000000000..24d9878b3a9b
--- /dev/null
+++ b/.test-infra/jenkins/userid_test.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+#
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This script performs basic analytic of performance tests results.
+# It operates in two modes:
+# --mode=report - In this mode script iterates over list of BigQuery tables
and
+# analyses the data. This mode is intended to be run on a regulars basis,
e.g. daily.
+# Report will contain average tests execution time of given metric, its
comparison with
+# average calculated from historical data, recent standard deviation and
standard
+# deviation calculated based on historical data.
+# --mode=validation - In this mode script will analyse single BigQuery table
and check
+# recent results.
+#
+# Other parameters are described in script. Notification is optional
parameter.
+# --send_notification - if present, script will send notification to slack
channel.
+# Requires setting env variable SLACK_WEBOOK_URL which value could be
obtained by
+# creating incoming webhook on Slack.
+#
+# This script is intended to be used only by Jenkins.
+# Example script usage:
+# verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import os
+import pwd
+
+print 'aaa'
+print os.getuid()
+print pwd.getpwall()
+print 'bbb'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 171188)
Time Spent: 23h (was: 22h 50m)
> Get Jenkins agents dockerized
> ------------------------------
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
> Reporter: yifan zou
> Assignee: yifan zou
> Priority: Major
> Time Spent: 23h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
