[
https://issues.apache.org/jira/browse/MESOS-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989680#comment-15989680
]
Joseph Wu commented on MESOS-7436:
----------------------------------
Exit code 137 in a docker container usually means your process is getting OOM
killed. Try monitoring the memory usage of processes inside the container and
consider increasing the amount of memory given to the container.
> Running on Mesos the Jenkins and Jest together crash silently Jenkins with
> code 137.
> ------------------------------------------------------------------------------------
>
> Key: MESOS-7436
> URL: https://issues.apache.org/jira/browse/MESOS-7436
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.28.1
> Environment: CPU
> Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
> Cores : 16
> Cache : 12288KB
> RAM
> 2x 16384MB
> Disks
> 2 x 2000 GB
> Motherboard
> X10SDV-TLN4F
> Kernel version
> 3.14.32-xxxx-grs-ipv6-64
> node version 7.9.0
> Reporter: Dimitri
>
> I apologize for the tutorial, and would like to thanks in advance the person
> who will achieve it.
> I am confident that test will crash in your environment, there is no reason
> to.
> In case you just wan't to see the crash , I have recorded it with my
> smartphone, it's not a professional footage but you can see it :
> https://drive.google.com/file/d/0B6drRJ3_BeQSMVh2RWE4cnc2NUU/view
> Prerequesite
> ==========
> In order to be exactly with the same configuration, you will need Host 32H
> offer at OVH.
> ```
> CPU
> Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz
> Cores : 16
> Cache : 12288KB
> RAM
> 2x 16384MB
> Disks
> 2 x 2000 GB
> Motherboard
> X10SDV-TLN4F
> Kernel version
> 3.14.32-xxxx-grs-ipv6-64
> ```
> 1. Install Debian Jessie. (can be skipped but potentially you won't get the
> error)
> 1. Install Mesos 0.28.1/Marathon 1.1.1. (can be skipped but potentially you
> won't get the error)
> 1. Install docker-engine 1.13.1. (can be skipped but potentially you won't
> get the error)
> During my test, I use the following configuration with Debian Jessie 8.0:
> Jenkins installation
> ===============
> 1. Download and run jenkins official image. From a separate bash: (you must
> have port 8080 available)
> docker run -p 8080:8080 jenkins
> 2. You will see in your terminal password. Copy it.
> Jenkins initial setup is required. An admin user has been created and a
> password generated.
> Please use the following password to proceed to installation:
> 39ae474fe5544c00881167cb8764cb14
> This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
> Or this will print the content of `
> /var/jenkins_home/secrets/initialAdminPassword`:
> docker exec -ti $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
> $1}') cat /var/jenkins_home/secrets/initialAdminPassword
> 3. Now open chrome `http://localhost:8080/` and past the code
> 
> 4. Click on **Select plugin to install**
> 
> 5. Go to manage jenkins > manage plugin and complete the installation by
> checking for installation the following plugins :
> - NodeJS
> 
> - Git plugin
> 
> Validate the change
> 6. Wait for the download and installation of the jenkins plugins :
> 
> 
> 7. Configure the administrator account as follow
> 
> 8. You will arrive on the jenkins dashboard
> 
> 9. Click on "Manage Jenkins"
> 
> 10. Now click on "Global Tool Configuration"
> 
> 11. Configure a NodeJS version 7.9.0
> 
> 12. Click on "New Item"
> 
> 13. Name it and choose "Freestyle project"
> 
> 14. Under "Source Code Management", select the radio "git" and add the
> following repository url : `https://github.com/kopax/jest-crashing-jenkins`
> 
> 15. Under "Build Environment" check "provide Node & npm bin/ folder to path"
> and select the version "7.9.0" of NodeJS.
> 
> 16. Under "Build", click on "Execute shell"
> 
> 17. Enter "npm install" and "npm test", then click "Save"
> 
> 18. Now click on Build now
> 
> 19. There will be a new job added. You can click on it and then click on
> "Console Output"
> 
> 20. While running the build, you can use `htop` to view your memory usage
> (`sudo apt-get update && sudo apt-get install htop -y`)
> 
> 21. Also you should note the `CONTAINER_ID` so you will see when it crash
> docker ps | grep jenkins
> abf10c67f072 jenkins "/bin/tini -- /usr/lo"
> About an hour ago Up About an hour 0.0.0.0:8080->8080/tcp, 50000/tcp
> reverent_saha
> I wasn't able to reproduce the crash it except in my production environment.
> This is how my test fail in production :
> 
> Notice the Jobs id 48, because when Jenkins will come back, it will miss that
> ID and jenkins will do like nothing happened before.
> _Tips 1: If you wish to have a more verbose log level when doing logs of your
> container you can do the following withing the containers:_
> docker exec -it $(docker ps | grep jenkins | head -1 | awk -F ' '
> '{print $1}') cat > /var/jenkins_home/log.properties <<EOF
> handlers=java.util.logging.ConsoleHandler
> jenkins.level=FINEST
> java.util.logging.ConsoleHandler.level=FINEST
> EOF
> And then restart your container with :
> docker stop $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
> $1}')
> docker start $(docker ps | grep jenkins | head -1 | awk -F ' ' '{print
> $1}') --env
> JAVA_OPTS="-Djava.util.logging.config.file=/var/jenkins_home/log.properties"
> _Tips 2: If you want to upgrade jenkins to the latest version (see the
> warning), you can activate a hidden button "Upgrade jenkins automatically
> button". For that, you need to give the good permissions to
> `/usr/share/jenkins` within the container, this will make the button appear
> in the "Jenkins Configuration" page and you will then be able to upgrade
> Jenkins_
> docker exec -it --user root $(docker ps | grep jenkins | head -1 |
> awk -F ' ' '{print $1}') chown jenkins:jenkins -R /usr/share/jenkins
> Note that I am running my Jenkins container on Mesos/Marathon and I only have
> the issue here. As attachement this is the marathon deployement json I use
> for deploying my jenkins container:
> ```json
> {
> "id": "developers/jenkins",
> "cmd": null,
> "mem": 4096,
> "cpus": 2,
> "instances": 1,
> "container": {
> "type": "DOCKER",
> "docker": {
> "image": "jenkins:latest",
> "forcePullImage": true,
> "network": "BRIDGE",
> "portMappings": [
> {
> "containerPort": 8080,
> "hostPort": 0,
> "protocol": "tcp"
> }
> ]
> },
> "volumes": [
> {
> "containerPath": "/var/jenkins_home",
> "hostPath": "/srv/developers/jenkins/var/jenkins_home",
> "mode": "RW"
> },
> {
> "containerPath": "/var/run/docker.sock",
> "hostPath": "/var/run/docker.sock",
> "mode": "RW"
> },
> {
> "containerPath": "/usr/bin/docker",
> "hostPath": "/usr/bin/docker",
> "mode": "RW"
> },
> {
> "containerPath": "/lib/x86_64-linux-gnu/libapparmor.so.1",
> "hostPath": "/usr/lib/x86_64-linux-gnu/libapparmor.so.1",
> "mode": "RW"
> }
> ]
> },
> "uris": [
> "file:///etc/docker.tar.gz"
> ],
> "healthChecks": [
> {
> "portIndex": 0,
> "protocol": "TCP",
> "gracePeriodSeconds": 30,
> "intervalSeconds": 10,
> "timeoutSeconds": 30,
> "maxConsecutiveFailures": 3
> }
> ],
> "upgradeStrategy": {
> "minimumHealthCapacity": 1,
> "maximumOverCapacity": 0.1
> }
> }
> ```
> This is the error I have on mesos :
> ```
> processname:marathon groupname:marathon pid:19 channel:stdout
> [2017-04-28 20:21:48,630] INFO Received status update for task
> kopaxgroup_developers_jenkins-2.dc2b0461-2c4d-11e7-9190-92077ff7c44b:
> TASK_FAILED (Docker container run error: Container exited on error: exited
> with status 137)
> (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$a504fd7e:Thread-1442)
> ```
> I have tried to find why, but the only place where I can reproduce this error
> is on mesos. Any Idea ?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
