Dimitri created MESOS-7436:
------------------------------
Summary: Running on Mesos the Jenkins and Jest together crash
silently Jenkins with code 137.
Key: MESOS-7436
URL: https://issues.apache.org/jira/browse/MESOS-7436
Project: Mesos
Issue Type: Bug
Affects Versions: 0.28.1
Environment: CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64
node version 7.9.0
Reporter: Dimitri
I apologize for the tutorial, and would like to thanks in advance the person
who will achieve it.
I am confident that test will crash in your environment, there is no reason to.
In case you just wan't to see the crash , I have recorded it with my
smartphone, it's not a professional footage but you can see it :
https://drive.google.com/file/d/0B6drRJ3_BeQSMVh2RWE4cnc2NUU/view
Prerequesite
==========
In order to be exactly with the same configuration, you will need Host 32H
offer at OVH.
```
CPU
Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
Cores : 16
Cache : 12288KB
RAM
2x 16384MB
Disks
2 x 2000 GB
Motherboard
X10SDV-TLN4F
Kernel version
3.14.32-xxxx-grs-ipv6-64
```
1. Install Debian Jessie. (can be skipped but potentially you won't get the
error)
1. Install Mesos 0.28.1/Marathon 1.1.1. (can be skipped but potentially you
won't get the error)
1. Install docker-engine 1.13.1. (can be skipped but potentially you won't get
the error)
During my test, I use the following configuration with Debian Jessie 8.0:
Jenkins installation
===============
1. Download and run jenkins official image. From a separate bash: (you must
have port 8080 available)
docker run -p 8080:8080 jenkins
2. You will see in your terminal password. Copy it.
Jenkins initial setup is required. An admin user has been created and a
password generated.
Please use the following password to proceed to installation:
39ae474fe5544c00881167cb8764cb14
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
Or this will print the content of `
/var/jenkins_home/secrets/initialAdminPassword`:
docker exec -ti $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
$1}') cat /var/jenkins_home/secrets/initialAdminPassword
3. Now open chrome `http://localhost:8080/` and past the code
4. Click on **Select plugin to install**
5. Go to manage jenkins > manage plugin and complete the installation by
checking for installation the following plugins :
- NodeJS
- Git plugin
Validate the change
6. Wait for the download and installation of the jenkins plugins :
7. Configure the administrator account as follow
8. You will arrive on the jenkins dashboard
9. Click on "Manage Jenkins"
10. Now click on "Global Tool Configuration"
11. Configure a NodeJS version 7.9.0
12. Click on "New Item"
13. Name it and choose "Freestyle project"
14. Under "Source Code Management", select the radio "git" and add the
following repository url : `https://github.com/kopax/jest-crashing-jenkins`
15. Under "Build Environment" check "provide Node & npm bin/ folder to path"
and select the version "7.9.0" of NodeJS.
16. Under "Build", click on "Execute shell"
17. Enter "npm install" and "npm test", then click "Save"
18. Now click on Build now
19. There will be a new job added. You can click on it and then click on
"Console Output"
20. While running the build, you can use `htop` to view your memory usage
(`sudo apt-get update && sudo apt-get install htop -y`)
21. Also you should note the `CONTAINER_ID` so you will see when it crash
docker ps | grep jenkins
abf10c67f072 jenkins "/bin/tini -- /usr/lo" About
an hour ago Up About an hour 0.0.0.0:8080->8080/tcp, 50000/tcp
reverent_saha
I wasn't able to reproduce the crash it except in my production environment.
This is how my test fail in production :
Notice the Jobs id 48, because when Jenkins will come back, it will miss that
ID and jenkins will do like nothing happened before.
_Tips 1: If you wish to have a more verbose log level when doing logs of your
container you can do the following withing the containers:_
docker exec -it $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
$1}') cat > /var/jenkins_home/log.properties <<EOF
handlers=java.util.logging.ConsoleHandler
jenkins.level=FINEST
java.util.logging.ConsoleHandler.level=FINEST
EOF
And then restart your container with :
docker stop $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print $1}')
docker start $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
$1}') --env
JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties"
_Tips 2: If you want to upgrade jenkins to the latest version (see the
warning), you can activate a hidden button "Upgrade jenkins automatically
button". For that, you need to give the good permissions to
`/usr/share/jenkins` within the container, this will make the button appear in
the "Jenkins Configuration" page and you will then be able to upgrade Jenkins_
docker exec -it --user root $(docker ps | grep jenkins | head -1 | awk
-F ' ' '{print $1}') chown jenkins:jenkins -R /usr/share/jenkins
Note that I am running my Jenkins container on Mesos/Marathon and I only have
the issue here. As attachement this is the marathon deployement json I use for
deploying my jenkins container:
```json
{
"id": "developers/jenkins",
"cmd": null,
"mem": 4096,
"cpus": 2,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "jenkins:latest",
"forcePullImage": true,
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 8080,
"hostPort": 0,
"protocol": "tcp"
}
]
},
"volumes": [
{
"containerPath": "/var/jenkins_home",
"hostPath": "/srv/developers/jenkins/var/jenkins_home",
"mode": "RW"
},
{
"containerPath": "/var/run/docker.sock",
"hostPath": "/var/run/docker.sock",
"mode": "RW"
},
{
"containerPath": "/usr/bin/docker",
"hostPath": "/usr/bin/docker",
"mode": "RW"
},
{
"containerPath": "/lib/x86_64-linux-gnu/libapparmor.so.1",
"hostPath": "/usr/lib/x86_64-linux-gnu/libapparmor.so.1",
"mode": "RW"
}
]
},
"uris": [
"file:///etc/docker.tar.gz"
],
"healthChecks": [
{
"portIndex": 0,
"protocol": "TCP",
"gracePeriodSeconds": 30,
"intervalSeconds": 10,
"timeoutSeconds": 30,
"maxConsecutiveFailures": 3
}
],
"upgradeStrategy": {
"minimumHealthCapacity": 1,
"maximumOverCapacity": 0.1
}
}
```
This is the error I have on mesos :
```
processname:marathon groupname:marathon pid:19 channel:stdout
[2017-04-28 20:21:48,630] INFO Received status update for task
kopaxgroup_developers_jenkins-2.dc2b0461-2c4d-11e7-9190-92077ff7c44b:
TASK_FAILED (Docker container run error: Container exited on error: exited with
status 137)
(mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$a504fd7e:Thread-1442)
```
I have tried to find why, but the only place where I can reproduce this error
is on mesos. Any Idea ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
