|
We've been experiencing problems with out builds for a while now and they've been pretty random.
Our builds create a sandbox within the workspace directory and use it to run tests and the sandbox is owned by root(as it's a whole system root). It's not a problem as long as all Jenkins does is fetch new changes from git to that workspace, but if the .git directory gets deleted when Jenkins tries doing a clone and it fails with:
Cloning the remote Git repository
Cloning repository g...@gitlab.something.net:something/something.git
ERROR: Failed to clean the workspace
java.nio.file.AccessDeniedException: /srv/jenkins/workspace/some-build-job/ec2-precise32-development/sandbox/jail/sys
The .git directory(along with .gitignore) has been disappearing randomly from workspace directories randomly and I've configured auditd service in order to find out which process has been deleting them, because I expected it to be something part of our process(Makefile?) and what I've found is stuff like this:
time->Sat Nov 22 08:32:03 2014
type=PATH msg=audit(1416645123.091:196): item=1 name="/srv/jenkins/workspace/xxxxxxx/label/ec2-precise32-development/.gitignore" inode=827146 dev=ca:01 mode=0100664 ouid=1000 ogid=1000 rdev=00:00
type=PATH msg=audit(1416645123.091:196): item=0 name="/srv/jenkins/workspace/xxxxxxx/label/ec2-precise32-development/" inode=814260 dev=ca:01 mode=040755 ouid=1000 ogid=1000 rdev=00:00
type=CWD msg=audit(1416645123.091:196): cwd="/home/ubuntu"
type=SYSCALL msg=audit(1416645123.091:196): arch=40000003 syscall=10 success=yes exit=0 a0=9f8be2f0 a1=b774c000 a2=b7709ff4 a3=9f8be2f0 items=2 ppid=1011 pid=4298 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=1 comm="java" exe="/usr/lib/jvm/jdk1.7.0_51/bin/java" key=(null)
A java process(Jenkins) executes syscall 10 (unlink/remove) on .gitignore(and I assume .git as well, I don't monitor it because the logs would be overflowing). I have no idea why it does that.
This particular deletion has caused a failure in a build four days later.
The interesting thing is that on the same slave around the same time (Sat Nov 22 08:32:03 2014) there was another build run, but it was completely different from the one that failed four days later. It used a different workspace directory, so it's hard to say if the two things are related.
This has been really messing up with out CI because the builds failing randomly are nightlies.
|
