The workspace cleanup isn't working - see the last bit of any recent ub1604 job: https://jenkins.impala.io/view/Utility/job/ubuntu-16.04-from-scratch/206/console
03:56:40.920 [WS-CLEANUP] Deleting project workspace...Cannot delete workspace :remote file operation failed: /home/ubuntu at hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c): java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. 03:56:48.161 ERROR: Step ‘Delete workspace when build is done’ failed: Cannot delete workspace: remote file operation failed: /home/ubuntu at hudson.remoting.Channel@4384d5b9:ubuntu-16.04 (i-032d527b9c801df4c): java.io.IOException: Unable to delete '/home/ubuntu'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. The workspace is $HOME, so you can't just delete it without being root. This could be changed to 1. A post-build script to "rm -rf ~/*". This doesn't reset everything, though - the job makes changes to other parts of the filesystem. 2. A post-build script to "sudo shutdown -h now" to make sure ec2 instances are not re-used. I'm not sure how Jenkins would feel about this. :-) 3. A post-build script to move $HOME to some archived location on the disk, to preserve debuggability. 4. A bash trap in the script to do one of the above. 5. Run the whole thing in a docker in the build machine, then delete the container when the script is done. Or don't, if there's enough disk space to not worry about that. 6. Do all of the work in a workspace inside $HOME. This would require some changes to bootstrap_development.sh. #5 is the most hermetic, I'd guess. On Thu, Aug 24, 2017 at 8:29 AM, Michael Brown <[email protected]> wrote: > Looks like someone has done this. > > On Wed, Aug 23, 2017 at 8:16 PM, Alexander Behm <[email protected]> > wrote: > >> Yes, let's please add the post-build action for sanity and consistency with >> our other jobs. >> >> On Wed, Aug 23, 2017 at 7:42 PM, Tim Armstrong <[email protected]> >> wrote: >> >> > Maybe the workspace just got left in a weird state - I think in most >> cases >> > "git init" followed by checking out a branch and doing a clean would >> work. >> > >> > Should we add the delete workspace post-build action? >> > >> > On Wed, Aug 23, 2017 at 5:32 PM, Michael Brown <[email protected]> >> wrote: >> > >> > > Not a known issue. I noticed ubuntu-16.04-from-scratch is not set to >> > clean >> > > up its workspace, and its config has not been touched since Aug 11. It >> > > seems strange we only saw this now.... >> > > >> > > On Wed, Aug 23, 2017 at 5:25 PM, Tim Armstrong < >> [email protected]> >> > > wrote: >> > > >> > > > Is this a known problem? My job failed because the Impala repo >> already >> > > > existed on the machine: >> > > > >> > > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/164/ >> > > > >> > > > *23:00:24* + /usr/bin/git init /home/ubuntu/Impala*23:00:24* >> > > > Reinitialized existing Git repository in /home/ubuntu/Impala/.git/ >> > > > .... >> > > > *23:02:18* + for ITER in '$(seq 1 10)'*23:02:18* + echo 'ATTEMPT: >> > > > 1'*23:02:18* ATTEMPT: 1*23:02:18* + /usr/bin/git checkout >> > > > FETCH_HEAD*23:02:18* + cat >> > > > /home/ubuntu/Impala/tmp.3tYBn0GUga*23:02:18* 23:02:18.712300 >> git.c:344 >> > > > trace: built-in: git 'checkout' 'FETCH_HEAD'*23:02:18* >> > > > error: The following untracked working tree files would be >> overwritten >> > > > by checkout:*23:02:18* .clang-format*23:02:18* >> > > > .clang-tidy*23:02:18* >> > > > .gitignore*23:02:18* CMakeLists.txt*23:02:18* >> > > > DISCLAIMER*23:02:18* >> > > > EXPORT_CONTROL.md*23:02:18* LICENSE.txt*23:02:18* >> > > > LOGS.md*23:02:18* >> > > > NOTICE.txt*23:02:18* README.md*23:02:18* >> > > > be/.gitignore*23:02:18* >> > > > be/.impala.doxy*23:02:18* be/CMakeLists.txt*23:02:18* >> > > > be/src/benchmarks/CMakeLists.txt*23:02:18* >> > > > be/src/benchmarks/atod-benchmark.cc*23:02:18* >> > > > be/src/benchmarks/atof-benchmark.cc*23:02:18* >> > > > be/src/benchmarks/atoi-benchmark.cc*23:02:18* >> > > > be/src/benchmarks/bit-packing-benchmark.cc*23:02:18* >> > > > be/src/benchmarks/bitmap-benchmark.cc >> > > > ... >> > > > >> > > >> > >>
