Hi, This is really crazy. The problem here is: The virtual machine is "suspended" when the node shuts down. After it comes back, of course the system time is wrong. As MacOSX does not have the VM client tools (it's not supported), after wakeup the VMM does not reset the wall time in the VM, so the whole thing relies on ntpd to reset the time - and then of course it jumps hard to new wall clock.
Unfortunately NTP does this very delayed, so the wall clock changes suddenly after approx. 5 minutes. Robert an me were expecting Solr tests to fail (because they depend on wall-clock not jumping), but funnily the whole JDK crashed this time. I have now changed the setup of the Jenkins slave to shutdown the VM completely, this will unfortunately take longer when it comes up again, but then always with a freshly booted OS. Currently I don't revert the harddisk to initial state, but as the node is killed hard (instead of ACPI shutdown), the file system may get corrupted. If this is the case, I can revert to snapshot, but I have to do this manually. In that case the workspaces and ivy cache would be empty again. I did this now (revert to snapshot). Alltogether it looks like the Darwin kernel does not do well in VMs... Maybe this is the reason why Apple officially disallows installing it inside a VM. If somebody has a real macintosh machine where we can connect to and run the salve, it would be fine. There is no special machine setup needed, only a separate user account (Jenkins) with SSH access and JDK 1.6, JDK 1.7 and python 3.2 installed at the standard Apple locations. The Master node automatically installs the Jenkins slave after connecting with SSH, so no setup needed. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: [email protected] > -----Original Message----- > From: Policeman Jenkins Server [mailto:[email protected]] > Sent: Sunday, December 30, 2012 1:10 PM > To: [email protected]; [email protected] > Subject: [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 21 - > Failure! > > Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/21/ > Java: 64bit/jdk1.7.0 -XX:+UseG1GC > > All tests passed > > Build Log: > [...truncated 723 lines...] > [junit4:junit4] ERROR: JVM J0 ended with an exception, command line: > /Library/Java/JavaVirtualMachines/jdk1.7.0_10.jdk/Contents/Home/jre/bin/ > java -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError - > XX:HeapDumpPath=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr- > trunk-MacOSX/heapdumps -Dtests.prefix=tests - > Dtests.seed=6BADAA10EB7E906B -Xmx512M -Dtests.iters= - > Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random - > Dtests.postingsformat=random -Dtests.locale=random - > Dtests.timezone=random -Dtests.directory=random - > Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=5.0 - > Dtests.cleanthreads=perMethod - > Djava.util.logging.config.file=/Users/jenkins/jenkins- > slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false - > Dtests.weekly=false -Dtests.slow=true -Dtests.asserts.gracious=false - > Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. - > Djunit4.tempDir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr- > trunk-MacOSX/lucene/build/core/test/temp - > Dclover.db.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/clover/db - > Djava.security.manager=org.apache.lucene.util.TestSecurityManager - > Djava.security.policy=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr- > trunk-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=5.0- > SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 - > Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory - > Djava.awt.headless=true -Dfile.encoding=UTF-8 -classpath > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/codecs/classes/java:/Users/jenkins/jenkins- > slave/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/test- > framework/classes/java:/Users/jenkins/jenkins-slave/workspace/Lucene- > Solr-trunk-MacOSX/lucene/test-framework/lib/junit- > 4.10.jar:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/test-framework/lib/randomizedtesting-runner- > 2.0.7.jar:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/core/classes/java:/Users/jenkins/jenkins- > slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/core/classes/test:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > launcher.jar:/Users/jenkins/.ant/lib/ivy-2.2.0.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > antlr.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > bcel.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > bsf.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > log4j.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > oro.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > regexp.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > resolver.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache- > xalan2.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons- > logging.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons- > net.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > jai.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > javamail.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > jdepend.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > jmf.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > jsch.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > junit.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > junit4.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > netrexx.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > swing.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant- > testutil.jar:/Users/jenkins/jenkins- > slave/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/Library/ > Java/JavaVirtualMachines/jdk1.7.0_10.jdk/Contents/Home/lib/tools.jar:/Us > ers/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4- > ant/jars/junit4-ant-2.0.7.jar -ea:org.apache.lucene... -ea:org.apache.solr... > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -flush -eventsfile > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/core/test/temp/junit4-J0- > 20121230_025221_076.events @/Users/jenkins/jenkins- > slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build/core/test/temp/junit4-J0- > 20121230_025221_076.suites > [junit4:junit4] ERROR: JVM J0 ended with an exception: Forked process > returned with error code: 134 Very likely a JVM crash. Process output piped > in logs above. > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1220) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:66) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:820) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:817) > [junit4:junit4] at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > [junit4:junit4] at > java.util.concurrent.FutureTask.run(FutureTask.java:166) > [junit4:junit4] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav > a:1110) > [junit4:junit4] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja > va:603) > [junit4:junit4] at java.lang.Thread.run(Thread.java:722) > > BUILD FAILED > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/build.xml:353: The following error occurred while executing this > line: > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/build.xml:39: The following error occurred while executing this line: > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/build.xml:50: The following error occurred while executing > this line: > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/common-build.xml:1176: The following error occurred while > executing this line: > /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-trunk- > MacOSX/lucene/common-build.xml:841: At least one slave process threw an > exception, first: Forked process returned with error code: 134 Very likely a > JVM crash. Process output piped in logs above. > > Total time: 558 minutes 35 seconds > Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording > test results Description set: Java: 64bit/jdk1.7.0 -XX:+UseG1GC Email was > triggered for: Failure Sending email for trigger: Failure > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
